Sep 08 04:00:25.968808 ip-10-0-133-37 systemd[1]: Starting Kubernetes Kubelet... Sep 08 04:00:26.336702 ip-10-0-133-37 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 04:00:26.336702 ip-10-0-133-37 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 08 04:00:26.336702 ip-10-0-133-37 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 04:00:26.336702 ip-10-0-133-37 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 08 04:00:26.336702 ip-10-0-133-37 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 04:00:26.337641 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.337533 2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 08 04:00:26.342993 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.342968 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 04:00:26.342993 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.342988 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 04:00:26.342993 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.342992 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 04:00:26.342993 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.342996 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 04:00:26.342993 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.342999 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 04:00:26.342993 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343002 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 04:00:26.342993 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343005 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 04:00:26.343250 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343008 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 04:00:26.343250 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343010 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 04:00:26.343250 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343013 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 04:00:26.343250 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343016 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 04:00:26.343250 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343019 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 04:00:26.343250 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343021 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 04:00:26.343250 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343024 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 04:00:26.343250 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343027 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 04:00:26.343250 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343029 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 04:00:26.343250 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343032 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 04:00:26.343250 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343034 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 04:00:26.343250 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343037 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 04:00:26.343250 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343039 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 04:00:26.343250 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343042 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 04:00:26.343250 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343044 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 04:00:26.343250 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343047 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 04:00:26.343250 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343050 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 04:00:26.343250 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343052 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 04:00:26.343250 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343054 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 04:00:26.343250 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343057 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 04:00:26.343726 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343063 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 04:00:26.343726 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343065 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 04:00:26.343726 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343068 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 04:00:26.343726 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343071 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 04:00:26.343726 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343073 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 04:00:26.343726 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343078 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 04:00:26.343726 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343083 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 04:00:26.343726 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343085 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 04:00:26.343726 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343088 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 04:00:26.343726 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343091 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 04:00:26.343726 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343094 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 04:00:26.343726 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343096 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 04:00:26.343726 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343099 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 04:00:26.343726 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343103 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 04:00:26.343726 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343106 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 04:00:26.343726 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343115 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 04:00:26.343726 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343118 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 04:00:26.343726 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343121 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 04:00:26.343726 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343124 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 08 04:00:26.343726 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343126 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 04:00:26.344202 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343129 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 04:00:26.344202 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343131 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 04:00:26.344202 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343135 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 04:00:26.344202 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343139 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 04:00:26.344202 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343141 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 04:00:26.344202 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343144 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 04:00:26.344202 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343146 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 04:00:26.344202 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343149 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 04:00:26.344202 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343152 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 04:00:26.344202 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343155 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 04:00:26.344202 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343157 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 04:00:26.344202 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343160 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 04:00:26.344202 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343162 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 04:00:26.344202 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343165 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 04:00:26.344202 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343167 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 04:00:26.344202 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343170 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 04:00:26.344202 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343172 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 04:00:26.344202 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343175 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 04:00:26.344202 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343177 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 04:00:26.344202 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343180 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 04:00:26.344670 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343182 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 04:00:26.344670 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343185 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 04:00:26.344670 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343188 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 04:00:26.344670 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343191 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 04:00:26.344670 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343196 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 04:00:26.344670 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343201 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 04:00:26.344670 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343204 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 04:00:26.344670 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343208 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 04:00:26.344670 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343211 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 04:00:26.344670 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343213 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 04:00:26.344670 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343216 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 04:00:26.344670 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343218 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 04:00:26.344670 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343222 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 04:00:26.344670 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343225 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 04:00:26.344670 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343228 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 04:00:26.344670 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343230 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 04:00:26.344670 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343233 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 04:00:26.344670 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343236 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 04:00:26.344670 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343238 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 04:00:26.345112 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343241 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 04:00:26.345112 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343243 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 04:00:26.345112 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343246 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 04:00:26.345112 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343249 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 04:00:26.345112 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343252 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 04:00:26.345112 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343254 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 04:00:26.345112 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343257 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 04:00:26.345112 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343259 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 04:00:26.345112 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343262 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 04:00:26.345112 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343265 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 04:00:26.345112 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343267 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 04:00:26.345112 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343270 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 04:00:26.345112 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.343272 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 04:00:26.345112 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344025 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 04:00:26.345112 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344033 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 04:00:26.345112 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344037 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 04:00:26.345112 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344040 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 04:00:26.345112 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344042 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 04:00:26.345112 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344046 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 04:00:26.345577 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344049 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 04:00:26.345577 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344054 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 04:00:26.345577 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344058 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 04:00:26.345577 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344061 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 04:00:26.345577 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344064 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 04:00:26.345577 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344066 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 04:00:26.345577 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344069 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 04:00:26.345577 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344072 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 04:00:26.345577 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344074 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 04:00:26.345577 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344077 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 04:00:26.345577 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344080 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 04:00:26.345577 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344083 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 04:00:26.345577 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344086 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 04:00:26.345577 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344088 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 08 04:00:26.345577 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344091 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 04:00:26.345577 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344093 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 04:00:26.345577 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344096 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 04:00:26.345577 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344099 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 04:00:26.345577 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344101 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 04:00:26.346028 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344103 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 04:00:26.346028 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344106 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 04:00:26.346028 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344108 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 04:00:26.346028 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344112 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 04:00:26.346028 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344115 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 04:00:26.346028 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344118 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 04:00:26.346028 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344121 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 04:00:26.346028 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344124 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 04:00:26.346028 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344126 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 04:00:26.346028 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344129 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 04:00:26.346028 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344131 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 04:00:26.346028 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344134 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 04:00:26.346028 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344137 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 04:00:26.346028 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344140 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 04:00:26.346028 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344142 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 04:00:26.346028 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344145 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 04:00:26.346028 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344147 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 04:00:26.346028 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344150 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 04:00:26.346028 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344153 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 04:00:26.346028 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344156 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 04:00:26.346521 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344158 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 04:00:26.346521 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344161 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 04:00:26.346521 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344163 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 04:00:26.346521 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344166 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 04:00:26.346521 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344168 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 04:00:26.346521 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344171 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 04:00:26.346521 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344174 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 04:00:26.346521 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344177 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 04:00:26.346521 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344179 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 04:00:26.346521 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344182 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 04:00:26.346521 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344185 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 04:00:26.346521 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344187 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 04:00:26.346521 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344190 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 04:00:26.346521 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344193 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 04:00:26.346521 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344195 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 04:00:26.346521 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344199 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 04:00:26.346521 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344202 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 04:00:26.346521 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344205 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 04:00:26.346521 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344208 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 04:00:26.346521 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344210 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 04:00:26.347096 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344213 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 04:00:26.347096 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344216 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 04:00:26.347096 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344219 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 04:00:26.347096 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344221 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 04:00:26.347096 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344224 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 04:00:26.347096 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344227 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 04:00:26.347096 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344230 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 04:00:26.347096 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344232 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 04:00:26.347096 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344235 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 04:00:26.347096 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344237 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 04:00:26.347096 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344240 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 04:00:26.347096 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344242 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 04:00:26.347096 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344245 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 04:00:26.347096 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344247 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 04:00:26.347096 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344249 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 04:00:26.347096 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344252 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 04:00:26.347096 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344254 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 04:00:26.347096 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344259 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 04:00:26.347096 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344263 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 04:00:26.347556 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344266 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 04:00:26.347556 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344269 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 04:00:26.347556 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344271 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 04:00:26.347556 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344274 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 04:00:26.347556 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344277 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 04:00:26.347556 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344281 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 04:00:26.347556 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344284 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 04:00:26.347556 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344287 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 04:00:26.347556 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344290 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 04:00:26.347556 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344293 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 04:00:26.347556 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344295 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 04:00:26.347556 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344298 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 04:00:26.347556 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344301 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 04:00:26.347556 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344303 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 04:00:26.347556 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344306 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 04:00:26.347556 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344699 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 04:00:26.347556 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344704 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 04:00:26.347556 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344707 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 04:00:26.347556 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344710 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 04:00:26.347556 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344713 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 04:00:26.348020 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344715 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 04:00:26.348020 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344718 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 04:00:26.348020 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344720 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 04:00:26.348020 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344723 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 04:00:26.348020 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344725 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 04:00:26.348020 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344728 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 04:00:26.348020 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344731 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 04:00:26.348020 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344734 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 04:00:26.348020 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344736 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 04:00:26.348020 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344738 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 04:00:26.348020 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344741 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 04:00:26.348020 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344744 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 04:00:26.348020 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344747 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 04:00:26.348020 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344749 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 04:00:26.348020 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344752 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 04:00:26.348020 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344755 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 04:00:26.348020 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344759 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 04:00:26.348020 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344763 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 04:00:26.348020 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344766 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 04:00:26.348020 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344768 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 04:00:26.348526 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344772 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 04:00:26.348526 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344774 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 04:00:26.348526 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344777 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 04:00:26.348526 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344779 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 04:00:26.348526 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344782 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 04:00:26.348526 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344785 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 04:00:26.348526 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344788 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 04:00:26.348526 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344791 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 04:00:26.348526 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344793 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 04:00:26.348526 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344796 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 04:00:26.348526 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344798 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 04:00:26.348526 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344801 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 04:00:26.348526 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344803 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 04:00:26.348526 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344806 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 04:00:26.348526 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344808 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 04:00:26.348526 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344811 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 04:00:26.348526 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344813 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 04:00:26.348526 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344815 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 04:00:26.348526 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344818 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 04:00:26.348526 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344820 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 04:00:26.348991 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344823 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 04:00:26.348991 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344825 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 04:00:26.348991 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344828 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 04:00:26.348991 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344831 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 04:00:26.348991 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344835 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 04:00:26.348991 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344838 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 04:00:26.348991 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344841 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 04:00:26.348991 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344844 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 04:00:26.348991 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344847 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 04:00:26.348991 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344850 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 04:00:26.348991 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344853 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 08 04:00:26.348991 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344856 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 04:00:26.348991 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344859 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 04:00:26.348991 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344862 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 04:00:26.348991 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344864 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 04:00:26.348991 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344867 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 04:00:26.348991 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344869 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 04:00:26.348991 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344871 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 04:00:26.348991 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344874 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 04:00:26.348991 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344876 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 04:00:26.349460 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344879 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 04:00:26.349460 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344881 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 04:00:26.349460 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344884 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 04:00:26.349460 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344886 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 04:00:26.349460 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344889 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 04:00:26.349460 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344892 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 04:00:26.349460 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344894 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 04:00:26.349460 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344897 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 04:00:26.349460 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344899 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 04:00:26.349460 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344901 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 04:00:26.349460 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344904 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 04:00:26.349460 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344906 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 04:00:26.349460 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344909 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 04:00:26.349460 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344911 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 04:00:26.349460 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344913 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 04:00:26.349460 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344916 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 04:00:26.349460 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344918 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 04:00:26.349460 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344921 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 04:00:26.349460 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344924 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 04:00:26.349947 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344927 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 04:00:26.349947 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344931 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 04:00:26.349947 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344934 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 04:00:26.349947 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344937 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 04:00:26.349947 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344942 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 04:00:26.349947 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344944 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 04:00:26.349947 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344947 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 04:00:26.349947 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344950 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 04:00:26.349947 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344952 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 04:00:26.349947 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344955 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 04:00:26.349947 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344957 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 04:00:26.349947 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344960 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 04:00:26.349947 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344962 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 04:00:26.349947 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344965 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 04:00:26.349947 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.344967 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 04:00:26.349947 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345043 2574 flags.go:64] FLAG: --address="0.0.0.0" Sep 08 04:00:26.349947 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345078 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 08 04:00:26.349947 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345085 2574 flags.go:64] FLAG: --anonymous-auth="true" Sep 08 04:00:26.349947 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345089 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 08 04:00:26.349947 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345093 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 08 04:00:26.349947 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345097 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 08 04:00:26.350430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345101 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 08 04:00:26.350430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345105 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 08 04:00:26.350430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345108 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 08 04:00:26.350430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345111 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 08 04:00:26.350430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345115 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 08 04:00:26.350430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345119 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 08 04:00:26.350430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345123 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 08 04:00:26.350430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345126 2574 flags.go:64] FLAG: --cgroup-root="" Sep 08 04:00:26.350430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345128 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 08 04:00:26.350430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345131 2574 flags.go:64] FLAG: --client-ca-file="" Sep 08 04:00:26.350430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345135 2574 flags.go:64] FLAG: --cloud-provider="external" Sep 08 04:00:26.350430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345138 2574 flags.go:64] FLAG: --cluster-dns="[]" Sep 08 04:00:26.350430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345142 2574 flags.go:64] FLAG: --cluster-domain="" Sep 08 04:00:26.350430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345145 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 08 04:00:26.350430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345148 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 08 04:00:26.350430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345153 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 08 04:00:26.350430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345156 2574 flags.go:64] FLAG: --container-log-max-files="5" Sep 08 04:00:26.350430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345160 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 08 04:00:26.350430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345164 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 08 04:00:26.350430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345167 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 08 04:00:26.350430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345170 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 08 04:00:26.350430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345173 2574 flags.go:64] FLAG: --contention-profiling="false" Sep 08 04:00:26.350430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345176 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 08 04:00:26.350430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345179 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 08 04:00:26.351013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345182 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 08 04:00:26.351013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345185 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 08 04:00:26.351013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345189 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 08 04:00:26.351013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345193 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 08 04:00:26.351013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345195 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 08 04:00:26.351013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345198 2574 flags.go:64] FLAG: --enable-load-reader="false" Sep 08 04:00:26.351013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345201 2574 flags.go:64] FLAG: --enable-server="true" Sep 08 04:00:26.351013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345204 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 08 04:00:26.351013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345209 2574 flags.go:64] FLAG: --event-burst="100" Sep 08 04:00:26.351013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345212 2574 flags.go:64] FLAG: --event-qps="50" Sep 08 04:00:26.351013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345215 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 08 04:00:26.351013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345218 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 08 04:00:26.351013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345220 2574 flags.go:64] FLAG: --eviction-hard="" Sep 08 04:00:26.351013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345226 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 08 04:00:26.351013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345229 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 08 04:00:26.351013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345232 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 08 04:00:26.351013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345235 2574 flags.go:64] FLAG: --eviction-soft="" Sep 08 04:00:26.351013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345238 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 08 04:00:26.351013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345240 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 08 04:00:26.351013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345244 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 08 04:00:26.351013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345247 2574 flags.go:64] FLAG: --experimental-mounter-path="" Sep 08 04:00:26.351013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345250 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 08 04:00:26.351013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345253 2574 flags.go:64] FLAG: --fail-swap-on="true" Sep 08 04:00:26.351013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345257 2574 flags.go:64] FLAG: --feature-gates="" Sep 08 04:00:26.351013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345261 2574 flags.go:64] FLAG: --file-check-frequency="20s" Sep 08 04:00:26.351616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345264 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 08 04:00:26.351616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345267 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 08 04:00:26.351616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345270 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 08 04:00:26.351616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345272 2574 flags.go:64] FLAG: --healthz-port="10248" Sep 08 04:00:26.351616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345276 2574 flags.go:64] FLAG: --help="false" Sep 08 04:00:26.351616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345278 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-133-37.ec2.internal" Sep 08 04:00:26.351616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345281 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 08 04:00:26.351616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345284 2574 flags.go:64] FLAG: --http-check-frequency="20s" Sep 08 04:00:26.351616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345287 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 08 04:00:26.351616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345291 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 08 04:00:26.351616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345294 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 08 04:00:26.351616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345297 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 08 04:00:26.351616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345299 2574 flags.go:64] FLAG: --image-service-endpoint="" Sep 08 04:00:26.351616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345302 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 08 04:00:26.351616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345305 2574 flags.go:64] FLAG: --kube-api-burst="100" Sep 08 04:00:26.351616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345308 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 08 04:00:26.351616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345311 2574 flags.go:64] FLAG: --kube-api-qps="50" Sep 08 04:00:26.351616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345313 2574 flags.go:64] FLAG: --kube-reserved="" Sep 08 04:00:26.351616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345316 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 08 04:00:26.351616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345319 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 08 04:00:26.351616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345325 2574 flags.go:64] FLAG: --kubelet-cgroups="" Sep 08 04:00:26.351616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345328 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 08 04:00:26.351616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345331 2574 flags.go:64] FLAG: --lock-file="" Sep 08 04:00:26.351616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345333 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 08 04:00:26.352190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345336 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 08 04:00:26.352190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345339 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 08 04:00:26.352190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345344 2574 flags.go:64] FLAG: --log-json-split-stream="false" Sep 08 04:00:26.352190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345348 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 08 04:00:26.352190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345351 2574 flags.go:64] FLAG: --log-text-split-stream="false" Sep 08 04:00:26.352190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345353 2574 flags.go:64] FLAG: --logging-format="text" Sep 08 04:00:26.352190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345358 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 08 04:00:26.352190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345361 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 08 04:00:26.352190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345364 2574 flags.go:64] FLAG: --manifest-url="" Sep 08 04:00:26.352190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345367 2574 flags.go:64] FLAG: --manifest-url-header="" Sep 08 04:00:26.352190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345371 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 08 04:00:26.352190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345374 2574 flags.go:64] FLAG: --max-open-files="1000000" Sep 08 04:00:26.352190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345383 2574 flags.go:64] FLAG: --max-pods="110" Sep 08 04:00:26.352190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345386 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 08 04:00:26.352190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345388 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 08 04:00:26.352190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345391 2574 flags.go:64] FLAG: --memory-manager-policy="None" Sep 08 04:00:26.352190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345394 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 08 04:00:26.352190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345397 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 08 04:00:26.352190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345400 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 08 04:00:26.352190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345403 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 08 04:00:26.352190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345410 2574 flags.go:64] FLAG: --node-status-max-images="50" Sep 08 04:00:26.352190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345414 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 08 04:00:26.352190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345416 2574 flags.go:64] FLAG: --oom-score-adj="-999" Sep 08 04:00:26.352190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345419 2574 flags.go:64] FLAG: --pod-cidr="" Sep 08 04:00:26.352768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345422 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 08 04:00:26.352768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345427 2574 flags.go:64] FLAG: --pod-manifest-path="" Sep 08 04:00:26.352768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345430 2574 flags.go:64] FLAG: --pod-max-pids="-1" Sep 08 04:00:26.352768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345433 2574 flags.go:64] FLAG: --pods-per-core="0" Sep 08 04:00:26.352768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345437 2574 flags.go:64] FLAG: --port="10250" Sep 08 04:00:26.352768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345440 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 08 04:00:26.352768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345443 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a260eece58702d28" Sep 08 04:00:26.352768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345446 2574 flags.go:64] FLAG: --qos-reserved="" Sep 08 04:00:26.352768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345449 2574 flags.go:64] FLAG: --read-only-port="10255" Sep 08 04:00:26.352768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345452 2574 flags.go:64] FLAG: --register-node="true" Sep 08 04:00:26.352768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345455 2574 flags.go:64] FLAG: --register-with-taints="" Sep 08 04:00:26.352768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345458 2574 flags.go:64] FLAG: --registry-burst="10" Sep 08 04:00:26.352768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345461 2574 flags.go:64] FLAG: --registry-qps="5" Sep 08 04:00:26.352768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345464 2574 flags.go:64] FLAG: --reserved-cpus="" Sep 08 04:00:26.352768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345467 2574 flags.go:64] FLAG: --reserved-memory="" Sep 08 04:00:26.352768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345472 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 08 04:00:26.352768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345475 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 08 04:00:26.352768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345478 2574 flags.go:64] FLAG: --rotate-certificates="false" Sep 08 04:00:26.352768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345481 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 08 04:00:26.352768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345484 2574 flags.go:64] FLAG: --runonce="false" Sep 08 04:00:26.352768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345486 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 08 04:00:26.352768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345489 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 08 04:00:26.352768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345492 2574 flags.go:64] FLAG: --seccomp-default="false" Sep 08 04:00:26.352768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345495 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 08 04:00:26.352768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345498 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 08 04:00:26.353386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345514 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 08 04:00:26.353386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345518 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 08 04:00:26.353386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345521 2574 flags.go:64] FLAG: --storage-driver-password="root" Sep 08 04:00:26.353386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345524 2574 flags.go:64] FLAG: --storage-driver-secure="false" Sep 08 04:00:26.353386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345527 2574 flags.go:64] FLAG: --storage-driver-table="stats" Sep 08 04:00:26.353386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345529 2574 flags.go:64] FLAG: --storage-driver-user="root" Sep 08 04:00:26.353386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345532 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 08 04:00:26.353386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345535 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 08 04:00:26.353386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345538 2574 flags.go:64] FLAG: --system-cgroups="" Sep 08 04:00:26.353386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345541 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 08 04:00:26.353386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345546 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 08 04:00:26.353386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345550 2574 flags.go:64] FLAG: --tls-cert-file="" Sep 08 04:00:26.353386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345552 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 08 04:00:26.353386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345557 2574 flags.go:64] FLAG: --tls-min-version="" Sep 08 04:00:26.353386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345560 2574 flags.go:64] FLAG: --tls-private-key-file="" Sep 08 04:00:26.353386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345563 2574 flags.go:64] FLAG: --topology-manager-policy="none" Sep 08 04:00:26.353386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345565 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 08 04:00:26.353386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345568 2574 flags.go:64] FLAG: --topology-manager-scope="container" Sep 08 04:00:26.353386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345571 2574 flags.go:64] FLAG: --v="2" Sep 08 04:00:26.353386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345575 2574 flags.go:64] FLAG: --version="false" Sep 08 04:00:26.353386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345580 2574 flags.go:64] FLAG: --vmodule="" Sep 08 04:00:26.353386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345584 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 08 04:00:26.353386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.345589 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 08 04:00:26.353386 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345718 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 04:00:26.353386 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345731 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 04:00:26.353986 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345734 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 04:00:26.353986 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345737 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 04:00:26.353986 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345740 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 04:00:26.353986 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345743 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 04:00:26.353986 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345745 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 04:00:26.353986 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345748 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 04:00:26.353986 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345750 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 04:00:26.353986 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345753 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 04:00:26.353986 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345755 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 04:00:26.353986 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345758 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 04:00:26.353986 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345760 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 04:00:26.353986 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345762 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 04:00:26.353986 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345765 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 08 04:00:26.353986 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345768 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 04:00:26.353986 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345770 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 04:00:26.353986 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345773 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 04:00:26.353986 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345776 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 04:00:26.353986 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345778 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 04:00:26.353986 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345782 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 04:00:26.353986 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345784 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 04:00:26.354456 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345787 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 04:00:26.354456 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345789 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 04:00:26.354456 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345792 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 04:00:26.354456 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345794 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 04:00:26.354456 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345797 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 04:00:26.354456 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345799 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 04:00:26.354456 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345802 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 04:00:26.354456 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345804 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 04:00:26.354456 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345807 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 04:00:26.354456 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345811 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 04:00:26.354456 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345813 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 04:00:26.354456 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345816 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 04:00:26.354456 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345818 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 04:00:26.354456 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345820 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 04:00:26.354456 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345823 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 04:00:26.354456 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345826 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 04:00:26.354456 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345828 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 04:00:26.354456 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345830 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 04:00:26.354456 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345833 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 04:00:26.354456 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345835 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 04:00:26.354951 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345838 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 04:00:26.354951 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345842 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 04:00:26.354951 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345846 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 04:00:26.354951 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345849 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 04:00:26.354951 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345852 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 04:00:26.354951 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345855 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 04:00:26.354951 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345857 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 04:00:26.354951 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345860 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 04:00:26.354951 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345862 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 04:00:26.354951 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345865 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 04:00:26.354951 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345869 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 04:00:26.354951 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345871 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 04:00:26.354951 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345874 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 04:00:26.354951 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345876 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 04:00:26.354951 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345879 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 04:00:26.354951 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345881 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 04:00:26.354951 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345884 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 04:00:26.354951 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345886 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 04:00:26.354951 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345889 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 04:00:26.354951 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345891 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 04:00:26.355413 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345894 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 04:00:26.355413 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345898 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 04:00:26.355413 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345901 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 04:00:26.355413 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345903 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 04:00:26.355413 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345906 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 04:00:26.355413 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345909 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 04:00:26.355413 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345912 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 04:00:26.355413 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345915 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 04:00:26.355413 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345917 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 04:00:26.355413 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345920 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 04:00:26.355413 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345922 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 04:00:26.355413 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345924 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 04:00:26.355413 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345927 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 04:00:26.355413 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345929 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 04:00:26.355413 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345932 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 04:00:26.355413 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345935 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 04:00:26.355413 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345937 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 04:00:26.355413 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345939 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 04:00:26.355413 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345942 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 04:00:26.355872 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345945 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 04:00:26.355872 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345947 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 04:00:26.355872 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345950 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 04:00:26.355872 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345953 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 04:00:26.355872 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345956 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 04:00:26.355872 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345959 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 04:00:26.355872 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345961 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 04:00:26.355872 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345964 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 04:00:26.355872 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345967 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 04:00:26.355872 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345970 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 04:00:26.355872 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345973 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 04:00:26.355872 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345976 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 04:00:26.355872 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345979 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 04:00:26.355872 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345982 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 04:00:26.355872 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345986 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 04:00:26.355872 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345989 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 04:00:26.355872 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345991 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 04:00:26.355872 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.345994 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 04:00:26.356307 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.346646 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 04:00:26.356307 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.353468 2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 08 04:00:26.356307 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.353486 2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 08 04:00:26.356307 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.354277 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 08 04:00:26.356307 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.354286 2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 08 04:00:26.356307 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354347 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 04:00:26.356307 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354352 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 04:00:26.356307 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354357 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 04:00:26.356307 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354361 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 04:00:26.356307 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354365 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 04:00:26.356307 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354369 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 04:00:26.356307 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354372 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 04:00:26.356307 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354375 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 04:00:26.356307 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354377 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 04:00:26.356685 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354380 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 04:00:26.356685 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354382 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 04:00:26.356685 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354385 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 04:00:26.356685 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354387 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 04:00:26.356685 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354390 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 04:00:26.356685 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354393 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 04:00:26.356685 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354396 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 04:00:26.356685 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354398 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 04:00:26.356685 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354401 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 04:00:26.356685 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354403 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 04:00:26.356685 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354406 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 04:00:26.356685 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354408 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 04:00:26.356685 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354411 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 04:00:26.356685 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354413 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 04:00:26.356685 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354416 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 04:00:26.356685 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354419 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 04:00:26.356685 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354421 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 04:00:26.356685 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354423 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 04:00:26.356685 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354426 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 04:00:26.356685 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354429 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 04:00:26.357160 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354433 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 04:00:26.357160 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354436 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 04:00:26.357160 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354439 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 04:00:26.357160 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354441 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 04:00:26.357160 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354444 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 04:00:26.357160 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354446 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 04:00:26.357160 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354449 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 04:00:26.357160 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354451 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 04:00:26.357160 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354454 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 04:00:26.357160 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354457 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 04:00:26.357160 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354459 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 04:00:26.357160 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354462 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 04:00:26.357160 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354465 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 04:00:26.357160 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354468 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 04:00:26.357160 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354470 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 04:00:26.357160 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354473 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 04:00:26.357160 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354476 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 04:00:26.357160 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354478 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 04:00:26.357160 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354481 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 04:00:26.357160 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354484 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 04:00:26.357644 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354486 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 04:00:26.357644 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354489 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 04:00:26.357644 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354491 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 04:00:26.357644 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354494 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 04:00:26.357644 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354496 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 04:00:26.357644 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354499 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 04:00:26.357644 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354515 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 04:00:26.357644 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354518 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 04:00:26.357644 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354521 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 04:00:26.357644 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354523 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 04:00:26.357644 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354526 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 04:00:26.357644 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354528 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 08 04:00:26.357644 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354532 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 04:00:26.357644 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354537 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 04:00:26.357644 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354540 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 04:00:26.357644 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354542 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 04:00:26.357644 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354545 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 04:00:26.357644 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354547 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 04:00:26.357644 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354550 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 04:00:26.357644 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354552 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 04:00:26.358105 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354555 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 04:00:26.358105 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354558 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 04:00:26.358105 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354560 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 04:00:26.358105 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354563 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 04:00:26.358105 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354565 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 04:00:26.358105 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354568 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 04:00:26.358105 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354570 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 04:00:26.358105 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354573 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 04:00:26.358105 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354575 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 04:00:26.358105 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354577 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 04:00:26.358105 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354580 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 04:00:26.358105 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354583 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 04:00:26.358105 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354585 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 04:00:26.358105 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354588 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 04:00:26.358105 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354590 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 04:00:26.358105 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354593 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 04:00:26.358105 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354595 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 04:00:26.358105 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354598 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 04:00:26.358105 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354600 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 04:00:26.358549 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354604 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 04:00:26.358549 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354608 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 04:00:26.358549 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354611 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 04:00:26.358549 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354614 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 04:00:26.358549 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354617 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 04:00:26.358549 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354621 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 04:00:26.358549 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354624 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 04:00:26.358549 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354628 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 04:00:26.358549 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354631 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 04:00:26.358549 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354634 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 04:00:26.358549 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354636 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 04:00:26.358549 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.354641 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 04:00:26.358549 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354764 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 04:00:26.358549 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354768 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 04:00:26.358549 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354771 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 04:00:26.358913 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354774 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 04:00:26.358913 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354777 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 04:00:26.358913 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354779 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 04:00:26.358913 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354782 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 04:00:26.358913 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354784 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 04:00:26.358913 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354787 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 04:00:26.358913 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354789 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 04:00:26.358913 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354792 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 04:00:26.358913 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354794 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 04:00:26.358913 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354797 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 04:00:26.358913 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354800 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 04:00:26.358913 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354804 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 04:00:26.358913 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354807 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 04:00:26.358913 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354810 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 04:00:26.358913 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354813 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 04:00:26.358913 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354816 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 04:00:26.358913 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354819 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 04:00:26.358913 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354821 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 04:00:26.358913 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354824 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 04:00:26.359355 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354826 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 04:00:26.359355 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354830 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 04:00:26.359355 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354832 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 04:00:26.359355 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354834 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 04:00:26.359355 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354837 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 04:00:26.359355 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354839 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 04:00:26.359355 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354843 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 04:00:26.359355 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354845 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 04:00:26.359355 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354848 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 04:00:26.359355 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354850 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 04:00:26.359355 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354853 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 04:00:26.359355 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354855 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 04:00:26.359355 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354858 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 04:00:26.359355 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354860 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 04:00:26.359355 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354863 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 04:00:26.359355 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354865 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 04:00:26.359355 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354868 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 04:00:26.359355 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354870 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 04:00:26.359355 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354872 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 04:00:26.359886 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354875 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 04:00:26.359886 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354877 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 08 04:00:26.359886 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354880 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 04:00:26.359886 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354882 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 04:00:26.359886 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354885 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 04:00:26.359886 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354887 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 04:00:26.359886 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354890 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 04:00:26.359886 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354892 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 04:00:26.359886 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354895 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 04:00:26.359886 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354897 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 04:00:26.359886 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354900 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 04:00:26.359886 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354902 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 04:00:26.359886 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354905 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 04:00:26.359886 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354907 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 04:00:26.359886 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354910 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 04:00:26.359886 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354912 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 04:00:26.359886 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354914 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 04:00:26.359886 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354918 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 04:00:26.359886 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354921 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 04:00:26.359886 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354923 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 04:00:26.360350 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354926 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 04:00:26.360350 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354928 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 04:00:26.360350 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354931 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 04:00:26.360350 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354933 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 04:00:26.360350 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354936 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 04:00:26.360350 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354938 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 04:00:26.360350 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354941 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 04:00:26.360350 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354943 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 04:00:26.360350 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354947 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 04:00:26.360350 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354950 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 04:00:26.360350 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354953 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 04:00:26.360350 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354955 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 04:00:26.360350 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354958 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 04:00:26.360350 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354962 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 04:00:26.360350 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354965 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 04:00:26.360350 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354967 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 04:00:26.360350 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354970 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 04:00:26.360350 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354972 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 04:00:26.360350 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354975 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 04:00:26.360803 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354977 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 04:00:26.360803 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354980 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 04:00:26.360803 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354982 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 04:00:26.360803 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354985 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 04:00:26.360803 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354988 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 04:00:26.360803 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354990 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 04:00:26.360803 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354992 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 04:00:26.360803 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354995 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 04:00:26.360803 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.354997 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 04:00:26.360803 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.355000 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 04:00:26.360803 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.355002 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 04:00:26.360803 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.355005 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 04:00:26.360803 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.355007 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 04:00:26.360803 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.355010 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 04:00:26.360803 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.355013 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 04:00:26.360803 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.355015 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 04:00:26.360803 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.355018 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 04:00:26.360803 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.355020 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 04:00:26.360803 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:26.355022 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 04:00:26.361241 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.355027 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 04:00:26.361241 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.355163 2574 server.go:962] "Client rotation is on, will bootstrap in background" Sep 08 04:00:26.361241 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.357254 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 08 04:00:26.361241 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.358099 2574 server.go:1019] "Starting client certificate rotation" Sep 08 04:00:26.361241 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.358196 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 04:00:26.361241 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.358233 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 04:00:26.361241 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.360363 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 04:00:26.367293 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.367272 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 04:00:26.376233 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.376215 2574 log.go:25] "Validated CRI v1 runtime API" Sep 08 04:00:26.379040 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.379025 2574 log.go:25] "Validated CRI v1 image API" Sep 08 04:00:26.379929 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.379908 2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 08 04:00:26.383764 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.383744 2574 fs.go:135] Filesystem UUIDs: map[0ac06067-8bfc-4cd2-9196-65eef502ba4b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ec83a2b9-e9da-48ef-b312-f00c49cb9860:/dev/nvme0n1p3] Sep 08 04:00:26.383822 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.383763 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 08 04:00:26.388641 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.388529 2574 manager.go:216] Machine: {Timestamp:2026-09-08 04:00:26.387349061 +0000 UTC m=+0.309809810 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3201060 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2eb32811943771aa8497cfa6d7f24d SystemUUID:ec2eb328-1194-3771-aa84-97cfa6d7f24d BootID:388fe841-1b5b-416e-9272-9a932d74a78c Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:df:61:bf:62:e7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:df:61:bf:62:e7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:0e:b3:4b:68:fc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 08 04:00:26.388641 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.388638 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 08 04:00:26.388779 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.388719 2574 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 08 04:00:26.391935 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.391904 2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 08 04:00:26.392090 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.391938 2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-37.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 08 04:00:26.392775 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.392765 2574 topology_manager.go:138] "Creating topology manager with none policy" Sep 08 04:00:26.392806 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.392777 2574 container_manager_linux.go:309] "Creating device plugin manager" Sep 08 04:00:26.392806 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.392785 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 04:00:26.393449 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.393439 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 04:00:26.394729 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.394719 2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 08 04:00:26.398733 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.398720 2574 state_mem.go:36] "Initialized new in-memory state store" Sep 08 04:00:26.398855 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.398845 2574 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 08 04:00:26.398884 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.398879 2574 kubelet.go:486] "Attempting to sync node with API server" Sep 08 04:00:26.398920 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.398890 2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 08 04:00:26.398920 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.398903 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 08 04:00:26.398920 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.398912 2574 kubelet.go:398] "Adding apiserver pod source" Sep 08 04:00:26.398920 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.398920 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 08 04:00:26.399312 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.399300 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 04:00:26.399357 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.399324 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 04:00:26.401418 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.401401 2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 08 04:00:26.402238 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.402219 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 04:00:26.403631 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.403615 2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 08 04:00:26.403717 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.403651 2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 08 04:00:26.403841 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.403828 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 08 04:00:26.403902 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.403847 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 08 04:00:26.403902 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.403856 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 08 04:00:26.403902 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.403864 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 08 04:00:26.403902 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.403875 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 08 04:00:26.403902 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.403890 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 08 04:00:26.404286 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.404276 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 08 04:00:26.404334 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.404289 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 08 04:00:26.404334 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.404299 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 08 04:00:26.404334 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.404309 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 08 04:00:26.404334 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.404332 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 08 04:00:26.404459 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.404344 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 08 04:00:26.405075 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.405064 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 08 04:00:26.405121 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.405078 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 08 04:00:26.409060 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.408926 2574 server.go:1268] "Started kubelet" Sep 08 04:00:26.412349 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.409017 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 08 04:00:26.412349 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.412256 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 08 04:00:26.412349 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.409003 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 08 04:00:26.412896 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.412748 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-37.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 08 04:00:26.413855 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:26.413828 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-37.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 08 04:00:26.414156 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.414124 2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 08 04:00:26.414263 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:26.414233 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 08 04:00:26.414700 ip-10-0-133-37 systemd[1]: Started Kubernetes Kubelet. Sep 08 04:00:26.414997 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.414979 2574 server.go:310] "Adding debug handlers to kubelet server" Sep 08 04:00:26.418101 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:26.418085 2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 08 04:00:26.418978 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.418963 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 08 04:00:26.419465 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.419450 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 08 04:00:26.419765 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:26.419745 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-37.ec2.internal\" not found" Sep 08 04:00:26.420181 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.420082 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 08 04:00:26.420181 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.420084 2574 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 08 04:00:26.420181 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.420124 2574 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 08 04:00:26.420341 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.420241 2574 reconstruct.go:97] "Volume reconstruction finished" Sep 08 04:00:26.420341 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.420249 2574 reconciler.go:29] "Reconciler: start to sync state" Sep 08 04:00:26.420614 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.420599 2574 factory.go:55] Registering systemd factory Sep 08 04:00:26.420666 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.420618 2574 factory.go:223] Registration of the systemd container factory successfully Sep 08 04:00:26.420831 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.420814 2574 factory.go:153] Registering CRI-O factory Sep 08 04:00:26.420831 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.420831 2574 factory.go:223] Registration of the crio container factory successfully Sep 08 04:00:26.420914 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.420878 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 08 04:00:26.420914 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.420898 2574 factory.go:103] Registering Raw factory Sep 08 04:00:26.420914 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.420911 2574 manager.go:1196] Started watching for new ooms in manager Sep 08 04:00:26.421250 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.421240 2574 manager.go:349] Starting recovery of all containers Sep 08 04:00:26.429437 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:26.429407 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 08 04:00:26.429964 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:26.429938 2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-37.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 08 04:00:26.430337 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:26.429467 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-37.ec2.internal.18d33c86be1a68df default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-37.ec2.internal,UID:ip-10-0-133-37.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-37.ec2.internal,},FirstTimestamp:2026-09-08 04:00:26.408904927 +0000 UTC m=+0.331365656,LastTimestamp:2026-09-08 04:00:26.408904927 +0000 UTC m=+0.331365656,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-37.ec2.internal,}" Sep 08 04:00:26.430430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.430340 2574 manager.go:354] Recovery completed Sep 08 04:00:26.436737 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.436724 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 04:00:26.438765 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.438735 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientMemory" Sep 08 04:00:26.438842 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.438832 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 04:00:26.438880 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.438849 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientPID" Sep 08 04:00:26.440101 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.439639 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 08 04:00:26.440101 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.439652 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 08 04:00:26.440101 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.439669 2574 state_mem.go:36] "Initialized new in-memory state store" Sep 08 04:00:26.440101 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.439922 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 04:00:26.441071 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.441054 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientMemory" Sep 08 04:00:26.441157 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.441091 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 04:00:26.441157 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.441106 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientPID" Sep 08 04:00:26.442038 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.442019 2574 policy_none.go:49] "None policy: Start" Sep 08 04:00:26.442117 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.442085 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 08 04:00:26.442117 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.442100 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 08 04:00:26.442334 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:26.442125 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-37.ec2.internal.18d33c86bfe203ac default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-37.ec2.internal,UID:ip-10-0-133-37.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-37.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-37.ec2.internal,},FirstTimestamp:2026-09-08 04:00:26.438763436 +0000 UTC m=+0.361224173,LastTimestamp:2026-09-08 04:00:26.438763436 +0000 UTC m=+0.361224173,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-37.ec2.internal,}" Sep 08 04:00:26.444096 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.444082 2574 policy_none.go:47] "Start" Sep 08 04:00:26.444186 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.444170 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vxnz2" Sep 08 04:00:26.452194 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:26.452111 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-37.ec2.internal.18d33c86bfe334c8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-37.ec2.internal,UID:ip-10-0-133-37.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-37.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-37.ec2.internal,},FirstTimestamp:2026-09-08 04:00:26.438841544 +0000 UTC m=+0.361302278,LastTimestamp:2026-09-08 04:00:26.438841544 +0000 UTC m=+0.361302278,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-37.ec2.internal,}" Sep 08 04:00:26.457225 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.457209 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vxnz2" Sep 08 04:00:26.471077 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.471062 2574 manager.go:337] "Starting Device Plugin manager" Sep 08 04:00:26.471178 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:26.471100 2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 08 04:00:26.471178 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.471110 2574 server.go:87] "Starting device plugin registration server" Sep 08 04:00:26.471457 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.471446 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 08 04:00:26.471514 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.471459 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 08 04:00:26.471709 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.471690 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 08 04:00:26.471806 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.471772 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 08 04:00:26.471806 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.471780 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 08 04:00:26.472452 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:26.472405 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 08 04:00:26.472452 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:26.472446 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-37.ec2.internal\" not found" Sep 08 04:00:26.560084 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.560043 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 08 04:00:26.561198 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.561184 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 08 04:00:26.561275 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.561242 2574 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 08 04:00:26.561275 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.561274 2574 kubelet.go:2448] "Starting kubelet main sync loop" Sep 08 04:00:26.561414 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:26.561317 2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 08 04:00:26.564039 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.564019 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 04:00:26.572042 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.572022 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 04:00:26.572956 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.572940 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientMemory" Sep 08 04:00:26.573050 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.572969 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 04:00:26.573050 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.572982 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientPID" Sep 08 04:00:26.573050 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.573005 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-37.ec2.internal" Sep 08 04:00:26.584422 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.584407 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-37.ec2.internal" Sep 08 04:00:26.584490 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:26.584431 2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-37.ec2.internal\": node \"ip-10-0-133-37.ec2.internal\" not found" Sep 08 04:00:26.610816 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:26.610728 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-37.ec2.internal\" not found" Sep 08 04:00:26.662195 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.662169 2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-37.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal"] Sep 08 04:00:26.662284 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.662240 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 04:00:26.663910 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.663894 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientMemory" Sep 08 04:00:26.663987 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.663923 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 04:00:26.663987 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.663934 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientPID" Sep 08 04:00:26.666156 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.666145 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 04:00:26.666277 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.666262 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-37.ec2.internal" Sep 08 04:00:26.666317 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.666298 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 04:00:26.666858 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.666839 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientMemory" Sep 08 04:00:26.666960 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.666864 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 04:00:26.666960 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.666874 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientPID" Sep 08 04:00:26.666960 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.666843 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientMemory" Sep 08 04:00:26.666960 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.666948 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 04:00:26.667137 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.666963 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientPID" Sep 08 04:00:26.671214 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.671199 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal" Sep 08 04:00:26.671287 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.671224 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 04:00:26.672344 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.671926 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientMemory" Sep 08 04:00:26.672344 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.671958 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 04:00:26.672344 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.671974 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientPID" Sep 08 04:00:26.681996 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:26.681978 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-37.ec2.internal\" not found" node="ip-10-0-133-37.ec2.internal" Sep 08 04:00:26.687279 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:26.687264 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-37.ec2.internal\" not found" node="ip-10-0-133-37.ec2.internal" Sep 08 04:00:26.711084 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:26.711068 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-37.ec2.internal\" not found" Sep 08 04:00:26.722407 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.722390 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/480faa79f347b524a5131c7d80428e3a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-37.ec2.internal\" (UID: \"480faa79f347b524a5131c7d80428e3a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-37.ec2.internal" Sep 08 04:00:26.722469 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.722412 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8da85c75381fab0440e1fd992aacf5e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal\" (UID: \"8da85c75381fab0440e1fd992aacf5e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal" Sep 08 04:00:26.722469 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.722440 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8da85c75381fab0440e1fd992aacf5e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal\" (UID: \"8da85c75381fab0440e1fd992aacf5e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal" Sep 08 04:00:26.812115 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:26.812085 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-37.ec2.internal\" not found" Sep 08 04:00:26.823388 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.823371 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/480faa79f347b524a5131c7d80428e3a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-37.ec2.internal\" (UID: \"480faa79f347b524a5131c7d80428e3a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-37.ec2.internal" Sep 08 04:00:26.823441 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.823395 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8da85c75381fab0440e1fd992aacf5e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal\" (UID: \"8da85c75381fab0440e1fd992aacf5e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal" Sep 08 04:00:26.823441 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.823411 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8da85c75381fab0440e1fd992aacf5e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal\" (UID: \"8da85c75381fab0440e1fd992aacf5e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal" Sep 08 04:00:26.823565 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.823482 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8da85c75381fab0440e1fd992aacf5e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal\" (UID: \"8da85c75381fab0440e1fd992aacf5e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal" Sep 08 04:00:26.823565 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.823481 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/480faa79f347b524a5131c7d80428e3a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-37.ec2.internal\" (UID: \"480faa79f347b524a5131c7d80428e3a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-37.ec2.internal" Sep 08 04:00:26.823565 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.823529 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8da85c75381fab0440e1fd992aacf5e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal\" (UID: \"8da85c75381fab0440e1fd992aacf5e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal" Sep 08 04:00:26.912630 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:26.912541 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-37.ec2.internal\" not found" Sep 08 04:00:26.984545 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.984493 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-37.ec2.internal" Sep 08 04:00:26.990323 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:26.990305 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal" Sep 08 04:00:27.012982 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:27.012959 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-37.ec2.internal\" not found" Sep 08 04:00:27.113522 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:27.113480 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-37.ec2.internal\" not found" Sep 08 04:00:27.214007 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:27.213946 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-37.ec2.internal\" not found" Sep 08 04:00:27.315195 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:27.315160 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-37.ec2.internal\" not found" Sep 08 04:00:27.359911 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:27.359887 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 08 04:00:27.360312 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:27.360044 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 04:00:27.395989 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:27.395968 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 04:00:27.415836 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:27.415807 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-37.ec2.internal\" not found" Sep 08 04:00:27.461423 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:27.461292 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 03:55:27 +0000 UTC" deadline="2028-05-20 04:21:41.052238416 +0000 UTC" Sep 08 04:00:27.461423 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:27.461328 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14880h21m13.590915315s" Sep 08 04:00:27.500094 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:27.500061 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 04:00:27.516842 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:27.516808 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-37.ec2.internal\" not found" Sep 08 04:00:27.617938 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:27.617891 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-37.ec2.internal\" not found" Sep 08 04:00:27.639297 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:27.639251 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod480faa79f347b524a5131c7d80428e3a.slice/crio-7695eb7d194705b32ffee22d11d285215d149518df5086a4b48bab459f00880e WatchSource:0}: Error finding container 7695eb7d194705b32ffee22d11d285215d149518df5086a4b48bab459f00880e: Status 404 returned error can't find the container with id 7695eb7d194705b32ffee22d11d285215d149518df5086a4b48bab459f00880e Sep 08 04:00:27.645802 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:27.645693 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 04:00:27.694145 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:27.694115 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 04:00:27.720236 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:27.720216 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal" Sep 08 04:00:27.735199 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:27.735182 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 08 04:00:27.736215 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:27.736203 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-37.ec2.internal" Sep 08 04:00:27.747834 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:27.747820 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 08 04:00:28.400076 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.400037 2574 apiserver.go:52] "Watching apiserver" Sep 08 04:00:28.409663 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.409639 2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 08 04:00:28.410076 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.410043 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-62m8t","kube-system/global-pull-secret-syncer-x4dkl","kube-system/kube-apiserver-proxy-ip-10-0-133-37.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt","openshift-image-registry/node-ca-qcbd8","openshift-network-diagnostics/network-check-target-l6l2z","openshift-ovn-kubernetes/ovnkube-node-86px5","kube-system/konnectivity-agent-84pxf","openshift-cluster-node-tuning-operator/tuned-hw7xq","openshift-dns/node-resolver-drx9g","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal","openshift-multus/multus-additional-cni-plugins-nrhb8","openshift-multus/multus-d95fg","openshift-multus/network-metrics-daemon-rzwvc"] Sep 08 04:00:28.412865 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.412843 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.415227 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.415207 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.416203 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.416185 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 08 04:00:28.416780 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.416760 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 08 04:00:28.416890 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.416792 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 08 04:00:28.416955 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.416898 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 08 04:00:28.417086 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.417066 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wp875\"" Sep 08 04:00:28.418108 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.418063 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 08 04:00:28.418108 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.418087 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 08 04:00:28.418334 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.418311 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:00:28.418410 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:28.418388 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzwvc" podUID="ff12a182-ed89-4513-938b-69971199ad25" Sep 08 04:00:28.418831 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.418814 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 08 04:00:28.418923 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.418814 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mkglr\"" Sep 08 04:00:28.419082 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.419069 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 08 04:00:28.419296 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.419278 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 08 04:00:28.419654 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.419607 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 08 04:00:28.420271 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.420172 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 08 04:00:28.420880 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.420863 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" Sep 08 04:00:28.423434 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.423417 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qcbd8" Sep 08 04:00:28.424276 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.424255 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 08 04:00:28.424370 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.424266 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 08 04:00:28.424432 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.424369 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 08 04:00:28.424838 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.424816 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t54rl\"" Sep 08 04:00:28.426014 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.425983 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 08 04:00:28.426107 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.426013 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:00:28.426166 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:28.426108 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6l2z" podUID="4c61ba14-8d9f-4634-9a0c-aac888042abe" Sep 08 04:00:28.426760 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.426733 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 08 04:00:28.426966 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.426947 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 08 04:00:28.427649 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.427624 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bxmpk\"" Sep 08 04:00:28.429926 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.429830 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.432219 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.432199 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-62m8t" Sep 08 04:00:28.432739 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.432717 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 08 04:00:28.432926 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.432806 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ksnjb\"" Sep 08 04:00:28.433244 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.433177 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 08 04:00:28.433244 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.433195 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-host-var-lib-cni-bin\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.433244 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.433232 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48zpv\" (UniqueName: \"kubernetes.io/projected/b11bc1d1-6ddb-4e21-8a0f-239e7407f358-kube-api-access-48zpv\") pod \"aws-ebs-csi-driver-node-h5xtt\" (UID: \"b11bc1d1-6ddb-4e21-8a0f-239e7407f358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" Sep 08 04:00:28.433459 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.433259 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-systemd-units\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.433459 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.433398 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-run-systemd\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.433459 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.433425 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-etc-openvswitch\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.433459 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.433451 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db9ph\" (UniqueName: \"kubernetes.io/projected/ff12a182-ed89-4513-938b-69971199ad25-kube-api-access-db9ph\") pod \"network-metrics-daemon-rzwvc\" (UID: \"ff12a182-ed89-4513-938b-69971199ad25\") " pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:00:28.433740 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.433475 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b11bc1d1-6ddb-4e21-8a0f-239e7407f358-socket-dir\") pod \"aws-ebs-csi-driver-node-h5xtt\" (UID: \"b11bc1d1-6ddb-4e21-8a0f-239e7407f358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" Sep 08 04:00:28.433740 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.433500 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b11bc1d1-6ddb-4e21-8a0f-239e7407f358-device-dir\") pod \"aws-ebs-csi-driver-node-h5xtt\" (UID: \"b11bc1d1-6ddb-4e21-8a0f-239e7407f358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" Sep 08 04:00:28.433740 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.433541 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b2959761-d748-49e4-8e37-c31e3c0517a8-env-overrides\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.433740 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.433565 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-host-run-k8s-cni-cncf-io\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.433740 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.433588 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-host-run-netns\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.433740 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.433613 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-etc-kubernetes\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.433740 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.433635 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b11bc1d1-6ddb-4e21-8a0f-239e7407f358-etc-selinux\") pod \"aws-ebs-csi-driver-node-h5xtt\" (UID: \"b11bc1d1-6ddb-4e21-8a0f-239e7407f358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" Sep 08 04:00:28.433740 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.433680 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b11bc1d1-6ddb-4e21-8a0f-239e7407f358-sys-fs\") pod \"aws-ebs-csi-driver-node-h5xtt\" (UID: \"b11bc1d1-6ddb-4e21-8a0f-239e7407f358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" Sep 08 04:00:28.433740 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.433707 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-host-kubelet\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.433740 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.433731 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-host-slash\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.434225 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.433753 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-var-lib-openvswitch\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.434225 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.433872 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-host-var-lib-kubelet\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.434225 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.433898 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-host-run-multus-certs\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.434225 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.433943 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zn6m\" (UniqueName: \"kubernetes.io/projected/ce867bba-0058-4825-bad0-6962d3ea4890-kube-api-access-9zn6m\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.434225 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.433976 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b11bc1d1-6ddb-4e21-8a0f-239e7407f358-registration-dir\") pod \"aws-ebs-csi-driver-node-h5xtt\" (UID: \"b11bc1d1-6ddb-4e21-8a0f-239e7407f358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" Sep 08 04:00:28.434225 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.434001 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-run-ovn\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.434225 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.434050 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce867bba-0058-4825-bad0-6962d3ea4890-cni-binary-copy\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.434225 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.434083 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-multus-socket-dir-parent\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.434225 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.434119 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-host-var-lib-cni-multus\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.434225 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.434154 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff12a182-ed89-4513-938b-69971199ad25-metrics-certs\") pod \"network-metrics-daemon-rzwvc\" (UID: \"ff12a182-ed89-4513-938b-69971199ad25\") " pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:00:28.434225 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.434185 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b11bc1d1-6ddb-4e21-8a0f-239e7407f358-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h5xtt\" (UID: \"b11bc1d1-6ddb-4e21-8a0f-239e7407f358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" Sep 08 04:00:28.434225 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.434207 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-host-run-netns\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.434924 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.434240 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-multus-cni-dir\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.434924 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.434304 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-multus-conf-dir\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.434924 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.434330 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-run-openvswitch\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.434924 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.434353 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-node-log\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.434924 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.434368 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-host-cni-netd\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.434924 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.434437 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b2959761-d748-49e4-8e37-c31e3c0517a8-ovnkube-config\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.434924 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.434460 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-system-cni-dir\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.434924 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.434483 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-cnibin\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.434924 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.434507 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-hostroot\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.434924 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.434543 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-log-socket\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.434924 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.434613 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-host-run-ovn-kubernetes\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.434924 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.434640 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-host-cni-bin\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.434924 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.434676 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc7v8\" (UniqueName: \"kubernetes.io/projected/b2959761-d748-49e4-8e37-c31e3c0517a8-kube-api-access-zc7v8\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.434924 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.434700 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ce867bba-0058-4825-bad0-6962d3ea4890-multus-daemon-config\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.434924 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.434718 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.434924 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.434737 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2959761-d748-49e4-8e37-c31e3c0517a8-ovn-node-metrics-cert\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.434924 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.434760 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b2959761-d748-49e4-8e37-c31e3c0517a8-ovnkube-script-lib\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.435833 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.434915 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:28.435833 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.434924 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 08 04:00:28.435833 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.434989 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cc2z2\"" Sep 08 04:00:28.435833 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:28.435026 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4dkl" podUID="c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb" Sep 08 04:00:28.435833 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.435228 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 08 04:00:28.435833 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.435313 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 08 04:00:28.435833 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.435653 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 04:00:28.437416 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.437397 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-drx9g" Sep 08 04:00:28.439865 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.439847 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-84pxf" Sep 08 04:00:28.440278 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.440255 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 08 04:00:28.440278 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.440271 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 08 04:00:28.440420 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.440346 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vkdrf\"" Sep 08 04:00:28.442466 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.442445 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nrhb8" Sep 08 04:00:28.442466 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.442461 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 08 04:00:28.442630 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.442473 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 08 04:00:28.442630 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.442626 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vsdrm\"" Sep 08 04:00:28.444806 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.444788 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 08 04:00:28.445124 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.445103 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rgnxt\"" Sep 08 04:00:28.445291 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.445267 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 08 04:00:28.461033 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.461013 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jr84m" Sep 08 04:00:28.470875 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.470855 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jr84m" Sep 08 04:00:28.521436 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.521413 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 08 04:00:28.535955 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.535918 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-run-openvswitch\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.536079 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.535990 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-node-log\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.536079 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.536064 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-run-openvswitch\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.536194 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.536107 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-original-pull-secret\") pod \"global-pull-secret-syncer-x4dkl\" (UID: \"c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb\") " pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:28.536194 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.536156 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkkh5\" (UniqueName: \"kubernetes.io/projected/f1262da6-55b5-49dd-8b8b-187c86dc46f6-kube-api-access-hkkh5\") pod \"node-resolver-drx9g\" (UID: \"f1262da6-55b5-49dd-8b8b-187c86dc46f6\") " pod="openshift-dns/node-resolver-drx9g" Sep 08 04:00:28.536194 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.536186 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-node-log\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.536360 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.536198 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-var-lib-kubelet\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.536360 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.536229 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-host-run-ovn-kubernetes\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.536360 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.536303 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc7v8\" (UniqueName: \"kubernetes.io/projected/b2959761-d748-49e4-8e37-c31e3c0517a8-kube-api-access-zc7v8\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.536360 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.536332 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-etc-modprobe-d\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.536595 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.536384 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.536595 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.536422 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b2959761-d748-49e4-8e37-c31e3c0517a8-ovnkube-script-lib\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.536595 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.536456 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-host-var-lib-cni-bin\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.536595 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.536481 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48zpv\" (UniqueName: \"kubernetes.io/projected/b11bc1d1-6ddb-4e21-8a0f-239e7407f358-kube-api-access-48zpv\") pod \"aws-ebs-csi-driver-node-h5xtt\" (UID: \"b11bc1d1-6ddb-4e21-8a0f-239e7407f358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" Sep 08 04:00:28.536595 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.536505 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz57w\" (UniqueName: \"kubernetes.io/projected/6698d58b-300e-4f04-a773-1756fcd6f558-kube-api-access-xz57w\") pod \"node-ca-qcbd8\" (UID: \"6698d58b-300e-4f04-a773-1756fcd6f558\") " pod="openshift-image-registry/node-ca-qcbd8" Sep 08 04:00:28.536595 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.536552 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-etc-sysctl-d\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.536909 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.536594 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-systemd-units\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.536909 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.536626 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.536909 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.536689 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-host-run-ovn-kubernetes\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.536909 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.536761 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b11bc1d1-6ddb-4e21-8a0f-239e7407f358-socket-dir\") pod \"aws-ebs-csi-driver-node-h5xtt\" (UID: \"b11bc1d1-6ddb-4e21-8a0f-239e7407f358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" Sep 08 04:00:28.536909 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.536784 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-systemd-units\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.536909 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.536818 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-run\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.536909 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.536887 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-host-var-lib-cni-bin\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.536909 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.536909 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-etc-kubernetes\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.537255 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.536936 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b11bc1d1-6ddb-4e21-8a0f-239e7407f358-sys-fs\") pod \"aws-ebs-csi-driver-node-h5xtt\" (UID: \"b11bc1d1-6ddb-4e21-8a0f-239e7407f358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" Sep 08 04:00:28.537255 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.536962 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-etc-sysctl-conf\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.537255 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.537001 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8ac44940-1e7f-48c1-a9db-8ac81a574f45-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nrhb8\" (UID: \"8ac44940-1e7f-48c1-a9db-8ac81a574f45\") " pod="openshift-multus/multus-additional-cni-plugins-nrhb8" Sep 08 04:00:28.537255 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.537074 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-etc-kubernetes\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.537255 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.537079 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-host-kubelet\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.537255 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.537142 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-host-kubelet\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.537255 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.537166 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b11bc1d1-6ddb-4e21-8a0f-239e7407f358-sys-fs\") pod \"aws-ebs-csi-driver-node-h5xtt\" (UID: \"b11bc1d1-6ddb-4e21-8a0f-239e7407f358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" Sep 08 04:00:28.537255 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.537183 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-host-slash\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.537615 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.537255 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-var-lib-openvswitch\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.537615 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.537283 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-host-slash\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.537842 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.537811 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b11bc1d1-6ddb-4e21-8a0f-239e7407f358-socket-dir\") pod \"aws-ebs-csi-driver-node-h5xtt\" (UID: \"b11bc1d1-6ddb-4e21-8a0f-239e7407f358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" Sep 08 04:00:28.537943 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.537910 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-var-lib-openvswitch\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.538112 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.538092 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b11bc1d1-6ddb-4e21-8a0f-239e7407f358-registration-dir\") pod \"aws-ebs-csi-driver-node-h5xtt\" (UID: \"b11bc1d1-6ddb-4e21-8a0f-239e7407f358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" Sep 08 04:00:28.538185 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.538147 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b2959761-d748-49e4-8e37-c31e3c0517a8-ovnkube-script-lib\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.538252 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.538163 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b11bc1d1-6ddb-4e21-8a0f-239e7407f358-registration-dir\") pod \"aws-ebs-csi-driver-node-h5xtt\" (UID: \"b11bc1d1-6ddb-4e21-8a0f-239e7407f358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" Sep 08 04:00:28.540560 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.540494 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c0f49eb0-c8d2-4383-a286-366557ec8894-iptables-alerter-script\") pod \"iptables-alerter-62m8t\" (UID: \"c0f49eb0-c8d2-4383-a286-366557ec8894\") " pod="openshift-network-operator/iptables-alerter-62m8t" Sep 08 04:00:28.540662 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.540596 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ac44940-1e7f-48c1-a9db-8ac81a574f45-system-cni-dir\") pod \"multus-additional-cni-plugins-nrhb8\" (UID: \"8ac44940-1e7f-48c1-a9db-8ac81a574f45\") " pod="openshift-multus/multus-additional-cni-plugins-nrhb8" Sep 08 04:00:28.540662 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.540639 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-run-ovn\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.540766 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.540675 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce867bba-0058-4825-bad0-6962d3ea4890-cni-binary-copy\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.540766 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.540740 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-run-ovn\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.540766 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.540743 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-multus-socket-dir-parent\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.540897 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.540801 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-host-var-lib-cni-multus\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.540897 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.540824 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-multus-socket-dir-parent\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.540985 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.540869 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff12a182-ed89-4513-938b-69971199ad25-metrics-certs\") pod \"network-metrics-daemon-rzwvc\" (UID: \"ff12a182-ed89-4513-938b-69971199ad25\") " pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:00:28.541308 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541283 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-host-var-lib-cni-multus\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.541397 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541286 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6698d58b-300e-4f04-a773-1756fcd6f558-serviceca\") pod \"node-ca-qcbd8\" (UID: \"6698d58b-300e-4f04-a773-1756fcd6f558\") " pod="openshift-image-registry/node-ca-qcbd8" Sep 08 04:00:28.541397 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541349 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-host\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.541397 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541377 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fab1c08f-5e53-42c6-a080-e510ae257469-konnectivity-ca\") pod \"konnectivity-agent-84pxf\" (UID: \"fab1c08f-5e53-42c6-a080-e510ae257469\") " pod="kube-system/konnectivity-agent-84pxf" Sep 08 04:00:28.541397 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:28.541385 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 04:00:28.541623 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541408 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-multus-cni-dir\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.541623 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541434 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fab1c08f-5e53-42c6-a080-e510ae257469-agent-certs\") pod \"konnectivity-agent-84pxf\" (UID: \"fab1c08f-5e53-42c6-a080-e510ae257469\") " pod="kube-system/konnectivity-agent-84pxf" Sep 08 04:00:28.541623 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:28.541472 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff12a182-ed89-4513-938b-69971199ad25-metrics-certs podName:ff12a182-ed89-4513-938b-69971199ad25 nodeName:}" failed. No retries permitted until 2026-09-08 04:00:29.041450785 +0000 UTC m=+2.963911520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff12a182-ed89-4513-938b-69971199ad25-metrics-certs") pod "network-metrics-daemon-rzwvc" (UID: "ff12a182-ed89-4513-938b-69971199ad25") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 04:00:28.541623 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541492 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-host-cni-netd\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.541623 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541498 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-multus-cni-dir\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.541623 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541519 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b2959761-d748-49e4-8e37-c31e3c0517a8-ovnkube-config\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.541623 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541571 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-host-cni-netd\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.541623 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541563 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-system-cni-dir\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.541949 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541628 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-cnibin\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.541949 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541648 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-hostroot\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.541949 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541690 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-system-cni-dir\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.541949 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541670 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwx6c\" (UniqueName: \"kubernetes.io/projected/c0f49eb0-c8d2-4383-a286-366557ec8894-kube-api-access-wwx6c\") pod \"iptables-alerter-62m8t\" (UID: \"c0f49eb0-c8d2-4383-a286-366557ec8894\") " pod="openshift-network-operator/iptables-alerter-62m8t" Sep 08 04:00:28.541949 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541736 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-log-socket\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.541949 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541739 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-hostroot\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.541949 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541753 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-cnibin\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.541949 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541770 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-host-cni-bin\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.541949 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541807 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ce867bba-0058-4825-bad0-6962d3ea4890-multus-daemon-config\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.541949 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541800 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-host-cni-bin\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.541949 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541838 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f1262da6-55b5-49dd-8b8b-187c86dc46f6-hosts-file\") pod \"node-resolver-drx9g\" (UID: \"f1262da6-55b5-49dd-8b8b-187c86dc46f6\") " pod="openshift-dns/node-resolver-drx9g" Sep 08 04:00:28.541949 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541863 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6698d58b-300e-4f04-a773-1756fcd6f558-host\") pod \"node-ca-qcbd8\" (UID: \"6698d58b-300e-4f04-a773-1756fcd6f558\") " pod="openshift-image-registry/node-ca-qcbd8" Sep 08 04:00:28.541949 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541869 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-log-socket\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.541949 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541887 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-etc-systemd\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.541949 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541926 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqhkn\" (UniqueName: \"kubernetes.io/projected/8ac44940-1e7f-48c1-a9db-8ac81a574f45-kube-api-access-bqhkn\") pod \"multus-additional-cni-plugins-nrhb8\" (UID: \"8ac44940-1e7f-48c1-a9db-8ac81a574f45\") " pod="openshift-multus/multus-additional-cni-plugins-nrhb8" Sep 08 04:00:28.541949 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541958 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2959761-d748-49e4-8e37-c31e3c0517a8-ovn-node-metrics-cert\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.542755 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.541984 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tns48\" (UniqueName: \"kubernetes.io/projected/4c61ba14-8d9f-4634-9a0c-aac888042abe-kube-api-access-tns48\") pod \"network-check-target-l6l2z\" (UID: \"4c61ba14-8d9f-4634-9a0c-aac888042abe\") " pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:00:28.542755 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542014 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-run-systemd\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.542755 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542037 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-etc-openvswitch\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.542755 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542054 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b2959761-d748-49e4-8e37-c31e3c0517a8-ovnkube-config\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.542755 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542062 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db9ph\" (UniqueName: \"kubernetes.io/projected/ff12a182-ed89-4513-938b-69971199ad25-kube-api-access-db9ph\") pod \"network-metrics-daemon-rzwvc\" (UID: \"ff12a182-ed89-4513-938b-69971199ad25\") " pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:00:28.542755 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542086 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b11bc1d1-6ddb-4e21-8a0f-239e7407f358-device-dir\") pod \"aws-ebs-csi-driver-node-h5xtt\" (UID: \"b11bc1d1-6ddb-4e21-8a0f-239e7407f358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" Sep 08 04:00:28.542755 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542099 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-run-systemd\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.542755 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542111 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1262da6-55b5-49dd-8b8b-187c86dc46f6-tmp-dir\") pod \"node-resolver-drx9g\" (UID: \"f1262da6-55b5-49dd-8b8b-187c86dc46f6\") " pod="openshift-dns/node-resolver-drx9g" Sep 08 04:00:28.542755 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542113 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-etc-openvswitch\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.542755 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542233 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-etc-sysconfig\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.542755 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542247 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b11bc1d1-6ddb-4e21-8a0f-239e7407f358-device-dir\") pod \"aws-ebs-csi-driver-node-h5xtt\" (UID: \"b11bc1d1-6ddb-4e21-8a0f-239e7407f358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" Sep 08 04:00:28.542755 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542258 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-etc-kubernetes\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.542755 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542287 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ac44940-1e7f-48c1-a9db-8ac81a574f45-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nrhb8\" (UID: \"8ac44940-1e7f-48c1-a9db-8ac81a574f45\") " pod="openshift-multus/multus-additional-cni-plugins-nrhb8" Sep 08 04:00:28.542755 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542298 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ce867bba-0058-4825-bad0-6962d3ea4890-multus-daemon-config\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.542755 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542319 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b2959761-d748-49e4-8e37-c31e3c0517a8-env-overrides\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.542755 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542340 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-host-run-k8s-cni-cncf-io\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.542755 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542360 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-host-run-netns\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.543485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542396 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b11bc1d1-6ddb-4e21-8a0f-239e7407f358-etc-selinux\") pod \"aws-ebs-csi-driver-node-h5xtt\" (UID: \"b11bc1d1-6ddb-4e21-8a0f-239e7407f358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" Sep 08 04:00:28.543485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542402 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-host-run-k8s-cni-cncf-io\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.543485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542422 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-lib-modules\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.543485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542433 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-host-run-netns\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.543485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542445 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3d167280-ff71-4e85-9abd-41b51f9358c6-etc-tuned\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.543485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542476 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ac44940-1e7f-48c1-a9db-8ac81a574f45-cnibin\") pod \"multus-additional-cni-plugins-nrhb8\" (UID: \"8ac44940-1e7f-48c1-a9db-8ac81a574f45\") " pod="openshift-multus/multus-additional-cni-plugins-nrhb8" Sep 08 04:00:28.543485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542500 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-host-var-lib-kubelet\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.543485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542539 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-host-run-multus-certs\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.543485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542543 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b11bc1d1-6ddb-4e21-8a0f-239e7407f358-etc-selinux\") pod \"aws-ebs-csi-driver-node-h5xtt\" (UID: \"b11bc1d1-6ddb-4e21-8a0f-239e7407f358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" Sep 08 04:00:28.543485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542572 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zn6m\" (UniqueName: \"kubernetes.io/projected/ce867bba-0058-4825-bad0-6962d3ea4890-kube-api-access-9zn6m\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.543485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542593 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0f49eb0-c8d2-4383-a286-366557ec8894-host-slash\") pod \"iptables-alerter-62m8t\" (UID: \"c0f49eb0-c8d2-4383-a286-366557ec8894\") " pod="openshift-network-operator/iptables-alerter-62m8t" Sep 08 04:00:28.543485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542586 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-host-run-multus-certs\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.543485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542616 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt4f6\" (UniqueName: \"kubernetes.io/projected/3d167280-ff71-4e85-9abd-41b51f9358c6-kube-api-access-lt4f6\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.543485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542644 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ac44940-1e7f-48c1-a9db-8ac81a574f45-cni-binary-copy\") pod \"multus-additional-cni-plugins-nrhb8\" (UID: \"8ac44940-1e7f-48c1-a9db-8ac81a574f45\") " pod="openshift-multus/multus-additional-cni-plugins-nrhb8" Sep 08 04:00:28.543485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542584 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-host-var-lib-kubelet\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.543485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542894 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-dbus\") pod \"global-pull-secret-syncer-x4dkl\" (UID: \"c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb\") " pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:28.543485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542656 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce867bba-0058-4825-bad0-6962d3ea4890-cni-binary-copy\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.544300 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542762 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b2959761-d748-49e4-8e37-c31e3c0517a8-env-overrides\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.544300 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542923 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b11bc1d1-6ddb-4e21-8a0f-239e7407f358-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h5xtt\" (UID: \"b11bc1d1-6ddb-4e21-8a0f-239e7407f358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" Sep 08 04:00:28.544300 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542709 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 08 04:00:28.544300 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542955 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-kubelet-config\") pod \"global-pull-secret-syncer-x4dkl\" (UID: \"c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb\") " pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:28.544300 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.542978 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d167280-ff71-4e85-9abd-41b51f9358c6-tmp\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.544300 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.543003 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-host-run-netns\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.544300 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.543016 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b11bc1d1-6ddb-4e21-8a0f-239e7407f358-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h5xtt\" (UID: \"b11bc1d1-6ddb-4e21-8a0f-239e7407f358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" Sep 08 04:00:28.544300 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.543039 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-multus-conf-dir\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.544300 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.543057 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2959761-d748-49e4-8e37-c31e3c0517a8-host-run-netns\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.544300 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.543088 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-sys\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.544300 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.543120 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce867bba-0058-4825-bad0-6962d3ea4890-multus-conf-dir\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.544300 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.543156 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8ac44940-1e7f-48c1-a9db-8ac81a574f45-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nrhb8\" (UID: \"8ac44940-1e7f-48c1-a9db-8ac81a574f45\") " pod="openshift-multus/multus-additional-cni-plugins-nrhb8" Sep 08 04:00:28.547633 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.547611 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc7v8\" (UniqueName: \"kubernetes.io/projected/b2959761-d748-49e4-8e37-c31e3c0517a8-kube-api-access-zc7v8\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.547725 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.547611 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48zpv\" (UniqueName: \"kubernetes.io/projected/b11bc1d1-6ddb-4e21-8a0f-239e7407f358-kube-api-access-48zpv\") pod \"aws-ebs-csi-driver-node-h5xtt\" (UID: \"b11bc1d1-6ddb-4e21-8a0f-239e7407f358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" Sep 08 04:00:28.548819 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.548795 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b2959761-d748-49e4-8e37-c31e3c0517a8-ovn-node-metrics-cert\") pod \"ovnkube-node-86px5\" (UID: \"b2959761-d748-49e4-8e37-c31e3c0517a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.556636 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.556613 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db9ph\" (UniqueName: \"kubernetes.io/projected/ff12a182-ed89-4513-938b-69971199ad25-kube-api-access-db9ph\") pod \"network-metrics-daemon-rzwvc\" (UID: \"ff12a182-ed89-4513-938b-69971199ad25\") " pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:00:28.556743 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.556649 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zn6m\" (UniqueName: \"kubernetes.io/projected/ce867bba-0058-4825-bad0-6962d3ea4890-kube-api-access-9zn6m\") pod \"multus-d95fg\" (UID: \"ce867bba-0058-4825-bad0-6962d3ea4890\") " pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.566478 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.566402 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-37.ec2.internal" event={"ID":"480faa79f347b524a5131c7d80428e3a","Type":"ContainerStarted","Data":"7695eb7d194705b32ffee22d11d285215d149518df5086a4b48bab459f00880e"} Sep 08 04:00:28.567631 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.567606 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal" event={"ID":"8da85c75381fab0440e1fd992aacf5e7","Type":"ContainerStarted","Data":"2d8d5f4b8c29ab261c24ab6143e97fbfa4f0e19d26d13737e08d75ff8c75b56b"} Sep 08 04:00:28.643718 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.643684 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz57w\" (UniqueName: \"kubernetes.io/projected/6698d58b-300e-4f04-a773-1756fcd6f558-kube-api-access-xz57w\") pod \"node-ca-qcbd8\" (UID: \"6698d58b-300e-4f04-a773-1756fcd6f558\") " pod="openshift-image-registry/node-ca-qcbd8" Sep 08 04:00:28.643718 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.643720 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-etc-sysctl-d\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.643945 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.643746 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-run\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.643945 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.643764 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-etc-sysctl-conf\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.643945 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.643787 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8ac44940-1e7f-48c1-a9db-8ac81a574f45-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nrhb8\" (UID: \"8ac44940-1e7f-48c1-a9db-8ac81a574f45\") " pod="openshift-multus/multus-additional-cni-plugins-nrhb8" Sep 08 04:00:28.643945 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.643806 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c0f49eb0-c8d2-4383-a286-366557ec8894-iptables-alerter-script\") pod \"iptables-alerter-62m8t\" (UID: \"c0f49eb0-c8d2-4383-a286-366557ec8894\") " pod="openshift-network-operator/iptables-alerter-62m8t" Sep 08 04:00:28.644145 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.643948 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-run\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.644145 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.643954 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-etc-sysctl-d\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.644145 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.643990 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ac44940-1e7f-48c1-a9db-8ac81a574f45-system-cni-dir\") pod \"multus-additional-cni-plugins-nrhb8\" (UID: \"8ac44940-1e7f-48c1-a9db-8ac81a574f45\") " pod="openshift-multus/multus-additional-cni-plugins-nrhb8" Sep 08 04:00:28.644145 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644022 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6698d58b-300e-4f04-a773-1756fcd6f558-serviceca\") pod \"node-ca-qcbd8\" (UID: \"6698d58b-300e-4f04-a773-1756fcd6f558\") " pod="openshift-image-registry/node-ca-qcbd8" Sep 08 04:00:28.644145 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644037 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-host\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.644145 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644054 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fab1c08f-5e53-42c6-a080-e510ae257469-konnectivity-ca\") pod \"konnectivity-agent-84pxf\" (UID: \"fab1c08f-5e53-42c6-a080-e510ae257469\") " pod="kube-system/konnectivity-agent-84pxf" Sep 08 04:00:28.644145 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644070 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fab1c08f-5e53-42c6-a080-e510ae257469-agent-certs\") pod \"konnectivity-agent-84pxf\" (UID: \"fab1c08f-5e53-42c6-a080-e510ae257469\") " pod="kube-system/konnectivity-agent-84pxf" Sep 08 04:00:28.644145 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644089 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwx6c\" (UniqueName: \"kubernetes.io/projected/c0f49eb0-c8d2-4383-a286-366557ec8894-kube-api-access-wwx6c\") pod \"iptables-alerter-62m8t\" (UID: \"c0f49eb0-c8d2-4383-a286-366557ec8894\") " pod="openshift-network-operator/iptables-alerter-62m8t" Sep 08 04:00:28.644145 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644091 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-etc-sysctl-conf\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.644145 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644106 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f1262da6-55b5-49dd-8b8b-187c86dc46f6-hosts-file\") pod \"node-resolver-drx9g\" (UID: \"f1262da6-55b5-49dd-8b8b-187c86dc46f6\") " pod="openshift-dns/node-resolver-drx9g" Sep 08 04:00:28.644145 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644124 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6698d58b-300e-4f04-a773-1756fcd6f558-host\") pod \"node-ca-qcbd8\" (UID: \"6698d58b-300e-4f04-a773-1756fcd6f558\") " pod="openshift-image-registry/node-ca-qcbd8" Sep 08 04:00:28.644145 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644138 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-etc-systemd\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.644145 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644143 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-host\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.644145 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644154 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqhkn\" (UniqueName: \"kubernetes.io/projected/8ac44940-1e7f-48c1-a9db-8ac81a574f45-kube-api-access-bqhkn\") pod \"multus-additional-cni-plugins-nrhb8\" (UID: \"8ac44940-1e7f-48c1-a9db-8ac81a574f45\") " pod="openshift-multus/multus-additional-cni-plugins-nrhb8" Sep 08 04:00:28.644888 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644173 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tns48\" (UniqueName: \"kubernetes.io/projected/4c61ba14-8d9f-4634-9a0c-aac888042abe-kube-api-access-tns48\") pod \"network-check-target-l6l2z\" (UID: \"4c61ba14-8d9f-4634-9a0c-aac888042abe\") " pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:00:28.644888 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644182 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ac44940-1e7f-48c1-a9db-8ac81a574f45-system-cni-dir\") pod \"multus-additional-cni-plugins-nrhb8\" (UID: \"8ac44940-1e7f-48c1-a9db-8ac81a574f45\") " pod="openshift-multus/multus-additional-cni-plugins-nrhb8" Sep 08 04:00:28.644888 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644193 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1262da6-55b5-49dd-8b8b-187c86dc46f6-tmp-dir\") pod \"node-resolver-drx9g\" (UID: \"f1262da6-55b5-49dd-8b8b-187c86dc46f6\") " pod="openshift-dns/node-resolver-drx9g" Sep 08 04:00:28.644888 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644223 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-etc-sysconfig\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.644888 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644238 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-etc-kubernetes\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.644888 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644253 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ac44940-1e7f-48c1-a9db-8ac81a574f45-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nrhb8\" (UID: \"8ac44940-1e7f-48c1-a9db-8ac81a574f45\") " pod="openshift-multus/multus-additional-cni-plugins-nrhb8" Sep 08 04:00:28.644888 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644276 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-lib-modules\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.644888 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644290 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3d167280-ff71-4e85-9abd-41b51f9358c6-etc-tuned\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.644888 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644304 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ac44940-1e7f-48c1-a9db-8ac81a574f45-cnibin\") pod \"multus-additional-cni-plugins-nrhb8\" (UID: \"8ac44940-1e7f-48c1-a9db-8ac81a574f45\") " pod="openshift-multus/multus-additional-cni-plugins-nrhb8" Sep 08 04:00:28.644888 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644322 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0f49eb0-c8d2-4383-a286-366557ec8894-host-slash\") pod \"iptables-alerter-62m8t\" (UID: \"c0f49eb0-c8d2-4383-a286-366557ec8894\") " pod="openshift-network-operator/iptables-alerter-62m8t" Sep 08 04:00:28.644888 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644336 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt4f6\" (UniqueName: \"kubernetes.io/projected/3d167280-ff71-4e85-9abd-41b51f9358c6-kube-api-access-lt4f6\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.644888 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644353 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ac44940-1e7f-48c1-a9db-8ac81a574f45-cni-binary-copy\") pod \"multus-additional-cni-plugins-nrhb8\" (UID: \"8ac44940-1e7f-48c1-a9db-8ac81a574f45\") " pod="openshift-multus/multus-additional-cni-plugins-nrhb8" Sep 08 04:00:28.644888 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644367 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-dbus\") pod \"global-pull-secret-syncer-x4dkl\" (UID: \"c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb\") " pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:28.644888 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644382 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-kubelet-config\") pod \"global-pull-secret-syncer-x4dkl\" (UID: \"c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb\") " pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:28.644888 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644396 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d167280-ff71-4e85-9abd-41b51f9358c6-tmp\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.644888 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644412 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-sys\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.644888 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644444 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8ac44940-1e7f-48c1-a9db-8ac81a574f45-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nrhb8\" (UID: \"8ac44940-1e7f-48c1-a9db-8ac81a574f45\") " pod="openshift-multus/multus-additional-cni-plugins-nrhb8" Sep 08 04:00:28.645560 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644443 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c0f49eb0-c8d2-4383-a286-366557ec8894-iptables-alerter-script\") pod \"iptables-alerter-62m8t\" (UID: \"c0f49eb0-c8d2-4383-a286-366557ec8894\") " pod="openshift-network-operator/iptables-alerter-62m8t" Sep 08 04:00:28.645560 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644489 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-original-pull-secret\") pod \"global-pull-secret-syncer-x4dkl\" (UID: \"c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb\") " pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:28.645560 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644505 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkkh5\" (UniqueName: \"kubernetes.io/projected/f1262da6-55b5-49dd-8b8b-187c86dc46f6-kube-api-access-hkkh5\") pod \"node-resolver-drx9g\" (UID: \"f1262da6-55b5-49dd-8b8b-187c86dc46f6\") " pod="openshift-dns/node-resolver-drx9g" Sep 08 04:00:28.645560 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644539 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-var-lib-kubelet\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.645560 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644584 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-etc-modprobe-d\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.645560 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644633 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6698d58b-300e-4f04-a773-1756fcd6f558-serviceca\") pod \"node-ca-qcbd8\" (UID: \"6698d58b-300e-4f04-a773-1756fcd6f558\") " pod="openshift-image-registry/node-ca-qcbd8" Sep 08 04:00:28.645560 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644647 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8ac44940-1e7f-48c1-a9db-8ac81a574f45-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nrhb8\" (UID: \"8ac44940-1e7f-48c1-a9db-8ac81a574f45\") " pod="openshift-multus/multus-additional-cni-plugins-nrhb8" Sep 08 04:00:28.645560 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644666 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-dbus\") pod \"global-pull-secret-syncer-x4dkl\" (UID: \"c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb\") " pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:28.645560 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644669 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ac44940-1e7f-48c1-a9db-8ac81a574f45-cnibin\") pod \"multus-additional-cni-plugins-nrhb8\" (UID: \"8ac44940-1e7f-48c1-a9db-8ac81a574f45\") " pod="openshift-multus/multus-additional-cni-plugins-nrhb8" Sep 08 04:00:28.645560 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644653 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ac44940-1e7f-48c1-a9db-8ac81a574f45-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nrhb8\" (UID: \"8ac44940-1e7f-48c1-a9db-8ac81a574f45\") " pod="openshift-multus/multus-additional-cni-plugins-nrhb8" Sep 08 04:00:28.645560 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644735 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-etc-modprobe-d\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.645560 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644746 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-kubelet-config\") pod \"global-pull-secret-syncer-x4dkl\" (UID: \"c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb\") " pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:28.645560 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:28.644827 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 04:00:28.645560 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.644833 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0f49eb0-c8d2-4383-a286-366557ec8894-host-slash\") pod \"iptables-alerter-62m8t\" (UID: \"c0f49eb0-c8d2-4383-a286-366557ec8894\") " pod="openshift-network-operator/iptables-alerter-62m8t" Sep 08 04:00:28.645560 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:28.644917 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-original-pull-secret podName:c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb nodeName:}" failed. No retries permitted until 2026-09-08 04:00:29.144897028 +0000 UTC m=+3.067357754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-original-pull-secret") pod "global-pull-secret-syncer-x4dkl" (UID: "c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb") : object "kube-system"/"original-pull-secret" not registered Sep 08 04:00:28.645560 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.645047 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6698d58b-300e-4f04-a773-1756fcd6f558-host\") pod \"node-ca-qcbd8\" (UID: \"6698d58b-300e-4f04-a773-1756fcd6f558\") " pod="openshift-image-registry/node-ca-qcbd8" Sep 08 04:00:28.645560 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.645098 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f1262da6-55b5-49dd-8b8b-187c86dc46f6-hosts-file\") pod \"node-resolver-drx9g\" (UID: \"f1262da6-55b5-49dd-8b8b-187c86dc46f6\") " pod="openshift-dns/node-resolver-drx9g" Sep 08 04:00:28.646401 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.645194 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-etc-systemd\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.646401 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.645254 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-var-lib-kubelet\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.646401 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.645293 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-sys\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.646401 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.645360 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-etc-sysconfig\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.646401 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.645442 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-etc-kubernetes\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.646401 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.645537 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ac44940-1e7f-48c1-a9db-8ac81a574f45-cni-binary-copy\") pod \"multus-additional-cni-plugins-nrhb8\" (UID: \"8ac44940-1e7f-48c1-a9db-8ac81a574f45\") " pod="openshift-multus/multus-additional-cni-plugins-nrhb8" Sep 08 04:00:28.646401 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.645729 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1262da6-55b5-49dd-8b8b-187c86dc46f6-tmp-dir\") pod \"node-resolver-drx9g\" (UID: \"f1262da6-55b5-49dd-8b8b-187c86dc46f6\") " pod="openshift-dns/node-resolver-drx9g" Sep 08 04:00:28.646401 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.645799 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8ac44940-1e7f-48c1-a9db-8ac81a574f45-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nrhb8\" (UID: \"8ac44940-1e7f-48c1-a9db-8ac81a574f45\") " pod="openshift-multus/multus-additional-cni-plugins-nrhb8" Sep 08 04:00:28.646401 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.645861 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d167280-ff71-4e85-9abd-41b51f9358c6-lib-modules\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.646401 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.646056 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fab1c08f-5e53-42c6-a080-e510ae257469-konnectivity-ca\") pod \"konnectivity-agent-84pxf\" (UID: \"fab1c08f-5e53-42c6-a080-e510ae257469\") " pod="kube-system/konnectivity-agent-84pxf" Sep 08 04:00:28.648122 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.648092 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fab1c08f-5e53-42c6-a080-e510ae257469-agent-certs\") pod \"konnectivity-agent-84pxf\" (UID: \"fab1c08f-5e53-42c6-a080-e510ae257469\") " pod="kube-system/konnectivity-agent-84pxf" Sep 08 04:00:28.648238 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.648081 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3d167280-ff71-4e85-9abd-41b51f9358c6-etc-tuned\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.648238 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.648203 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d167280-ff71-4e85-9abd-41b51f9358c6-tmp\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.659813 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.659750 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwx6c\" (UniqueName: \"kubernetes.io/projected/c0f49eb0-c8d2-4383-a286-366557ec8894-kube-api-access-wwx6c\") pod \"iptables-alerter-62m8t\" (UID: \"c0f49eb0-c8d2-4383-a286-366557ec8894\") " pod="openshift-network-operator/iptables-alerter-62m8t" Sep 08 04:00:28.662051 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.662025 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt4f6\" (UniqueName: \"kubernetes.io/projected/3d167280-ff71-4e85-9abd-41b51f9358c6-kube-api-access-lt4f6\") pod \"tuned-hw7xq\" (UID: \"3d167280-ff71-4e85-9abd-41b51f9358c6\") " pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.663266 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:28.663238 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 04:00:28.663358 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:28.663271 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 04:00:28.663358 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:28.663289 2574 projected.go:196] Error preparing data for projected volume kube-api-access-tns48 for pod openshift-network-diagnostics/network-check-target-l6l2z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 04:00:28.663477 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:28.663368 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c61ba14-8d9f-4634-9a0c-aac888042abe-kube-api-access-tns48 podName:4c61ba14-8d9f-4634-9a0c-aac888042abe nodeName:}" failed. No retries permitted until 2026-09-08 04:00:29.163348111 +0000 UTC m=+3.085808854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tns48" (UniqueName: "kubernetes.io/projected/4c61ba14-8d9f-4634-9a0c-aac888042abe-kube-api-access-tns48") pod "network-check-target-l6l2z" (UID: "4c61ba14-8d9f-4634-9a0c-aac888042abe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 04:00:28.664271 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.664238 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkkh5\" (UniqueName: \"kubernetes.io/projected/f1262da6-55b5-49dd-8b8b-187c86dc46f6-kube-api-access-hkkh5\") pod \"node-resolver-drx9g\" (UID: \"f1262da6-55b5-49dd-8b8b-187c86dc46f6\") " pod="openshift-dns/node-resolver-drx9g" Sep 08 04:00:28.665229 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.665206 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqhkn\" (UniqueName: \"kubernetes.io/projected/8ac44940-1e7f-48c1-a9db-8ac81a574f45-kube-api-access-bqhkn\") pod \"multus-additional-cni-plugins-nrhb8\" (UID: \"8ac44940-1e7f-48c1-a9db-8ac81a574f45\") " pod="openshift-multus/multus-additional-cni-plugins-nrhb8" Sep 08 04:00:28.665905 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.665883 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz57w\" (UniqueName: \"kubernetes.io/projected/6698d58b-300e-4f04-a773-1756fcd6f558-kube-api-access-xz57w\") pod \"node-ca-qcbd8\" (UID: \"6698d58b-300e-4f04-a773-1756fcd6f558\") " pod="openshift-image-registry/node-ca-qcbd8" Sep 08 04:00:28.725816 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.725780 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:28.734342 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.734318 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d95fg" Sep 08 04:00:28.743353 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.743331 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" Sep 08 04:00:28.749273 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.749227 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qcbd8" Sep 08 04:00:28.756086 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.756064 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" Sep 08 04:00:28.764105 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.764084 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-62m8t" Sep 08 04:00:28.770275 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.770256 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-drx9g" Sep 08 04:00:28.778160 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.778136 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-84pxf" Sep 08 04:00:28.783044 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.783024 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nrhb8" Sep 08 04:00:28.787954 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:28.787932 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 04:00:29.047821 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:29.047220 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff12a182-ed89-4513-938b-69971199ad25-metrics-certs\") pod \"network-metrics-daemon-rzwvc\" (UID: \"ff12a182-ed89-4513-938b-69971199ad25\") " pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:00:29.047821 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:29.047464 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 04:00:29.047821 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:29.047542 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff12a182-ed89-4513-938b-69971199ad25-metrics-certs podName:ff12a182-ed89-4513-938b-69971199ad25 nodeName:}" failed. No retries permitted until 2026-09-08 04:00:30.047522834 +0000 UTC m=+3.969983555 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff12a182-ed89-4513-938b-69971199ad25-metrics-certs") pod "network-metrics-daemon-rzwvc" (UID: "ff12a182-ed89-4513-938b-69971199ad25") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 04:00:29.073730 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:29.073697 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2959761_d748_49e4_8e37_c31e3c0517a8.slice/crio-6dd87d089f1b986fabb6eb59bac4938f07a28b3b20f64da1d3d80bce08f56da1 WatchSource:0}: Error finding container 6dd87d089f1b986fabb6eb59bac4938f07a28b3b20f64da1d3d80bce08f56da1: Status 404 returned error can't find the container with id 6dd87d089f1b986fabb6eb59bac4938f07a28b3b20f64da1d3d80bce08f56da1 Sep 08 04:00:29.074663 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:29.074629 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d167280_ff71_4e85_9abd_41b51f9358c6.slice/crio-0d8b7674a116ccf8e2d1fb0145ca53c42f4e5a814fda9b3b686b63d0910b67bd WatchSource:0}: Error finding container 0d8b7674a116ccf8e2d1fb0145ca53c42f4e5a814fda9b3b686b63d0910b67bd: Status 404 returned error can't find the container with id 0d8b7674a116ccf8e2d1fb0145ca53c42f4e5a814fda9b3b686b63d0910b67bd Sep 08 04:00:29.103746 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:29.103647 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce867bba_0058_4825_bad0_6962d3ea4890.slice/crio-d2b1deac807beff19e85a1613b0d1cd191abf1252f93e24cb2a024eccb9a802b WatchSource:0}: Error finding container d2b1deac807beff19e85a1613b0d1cd191abf1252f93e24cb2a024eccb9a802b: Status 404 returned error can't find the container with id d2b1deac807beff19e85a1613b0d1cd191abf1252f93e24cb2a024eccb9a802b Sep 08 04:00:29.105443 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:29.104706 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f49eb0_c8d2_4383_a286_366557ec8894.slice/crio-bf0535783cafe7969478e9195c40db9eac95cbbe2a05ad4af4a1402f305ca252 WatchSource:0}: Error finding container bf0535783cafe7969478e9195c40db9eac95cbbe2a05ad4af4a1402f305ca252: Status 404 returned error can't find the container with id bf0535783cafe7969478e9195c40db9eac95cbbe2a05ad4af4a1402f305ca252 Sep 08 04:00:29.105627 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:29.105580 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1262da6_55b5_49dd_8b8b_187c86dc46f6.slice/crio-b4924ac2795deb76c224d71be9dfd082cbdb803649b32431f1757fca4ff08bb3 WatchSource:0}: Error finding container b4924ac2795deb76c224d71be9dfd082cbdb803649b32431f1757fca4ff08bb3: Status 404 returned error can't find the container with id b4924ac2795deb76c224d71be9dfd082cbdb803649b32431f1757fca4ff08bb3 Sep 08 04:00:29.107106 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:29.106653 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6698d58b_300e_4f04_a773_1756fcd6f558.slice/crio-40109c53b717cea9ef2618ca527b20fc056f6561c883eb8c5b686bc873651b03 WatchSource:0}: Error finding container 40109c53b717cea9ef2618ca527b20fc056f6561c883eb8c5b686bc873651b03: Status 404 returned error can't find the container with id 40109c53b717cea9ef2618ca527b20fc056f6561c883eb8c5b686bc873651b03 Sep 08 04:00:29.108658 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:29.108226 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb11bc1d1_6ddb_4e21_8a0f_239e7407f358.slice/crio-a49510636176ba81801b6f76f3d8363d6ffc622118aa56150fff3ff447026b2b WatchSource:0}: Error finding container a49510636176ba81801b6f76f3d8363d6ffc622118aa56150fff3ff447026b2b: Status 404 returned error can't find the container with id a49510636176ba81801b6f76f3d8363d6ffc622118aa56150fff3ff447026b2b Sep 08 04:00:29.111674 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:00:29.111649 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac44940_1e7f_48c1_a9db_8ac81a574f45.slice/crio-7bacd546b953cad5712e9c9b1410713ea646650e7966cf3ba5524b62f4f18c4e WatchSource:0}: Error finding container 7bacd546b953cad5712e9c9b1410713ea646650e7966cf3ba5524b62f4f18c4e: Status 404 returned error can't find the container with id 7bacd546b953cad5712e9c9b1410713ea646650e7966cf3ba5524b62f4f18c4e Sep 08 04:00:29.148139 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:29.148116 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-original-pull-secret\") pod \"global-pull-secret-syncer-x4dkl\" (UID: \"c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb\") " pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:29.148307 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:29.148261 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 04:00:29.148423 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:29.148324 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-original-pull-secret podName:c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb nodeName:}" failed. No retries permitted until 2026-09-08 04:00:30.148306193 +0000 UTC m=+4.070766916 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-original-pull-secret") pod "global-pull-secret-syncer-x4dkl" (UID: "c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb") : object "kube-system"/"original-pull-secret" not registered Sep 08 04:00:29.249206 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:29.249165 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tns48\" (UniqueName: \"kubernetes.io/projected/4c61ba14-8d9f-4634-9a0c-aac888042abe-kube-api-access-tns48\") pod \"network-check-target-l6l2z\" (UID: \"4c61ba14-8d9f-4634-9a0c-aac888042abe\") " pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:00:29.249388 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:29.249364 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 04:00:29.249472 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:29.249394 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 04:00:29.249472 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:29.249406 2574 projected.go:196] Error preparing data for projected volume kube-api-access-tns48 for pod openshift-network-diagnostics/network-check-target-l6l2z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 04:00:29.249472 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:29.249466 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c61ba14-8d9f-4634-9a0c-aac888042abe-kube-api-access-tns48 podName:4c61ba14-8d9f-4634-9a0c-aac888042abe nodeName:}" failed. No retries permitted until 2026-09-08 04:00:30.249448411 +0000 UTC m=+4.171909155 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tns48" (UniqueName: "kubernetes.io/projected/4c61ba14-8d9f-4634-9a0c-aac888042abe-kube-api-access-tns48") pod "network-check-target-l6l2z" (UID: "4c61ba14-8d9f-4634-9a0c-aac888042abe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 04:00:29.474993 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:29.474920 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 03:55:28 +0000 UTC" deadline="2028-04-26 04:25:03.272764716 +0000 UTC" Sep 08 04:00:29.475503 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:29.474995 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14304h24m33.797775977s" Sep 08 04:00:29.561840 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:29.561805 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:00:29.561840 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:29.561837 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:29.562031 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:29.561940 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6l2z" podUID="4c61ba14-8d9f-4634-9a0c-aac888042abe" Sep 08 04:00:29.562187 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:29.562089 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4dkl" podUID="c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb" Sep 08 04:00:29.572246 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:29.572212 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrhb8" event={"ID":"8ac44940-1e7f-48c1-a9db-8ac81a574f45","Type":"ContainerStarted","Data":"7bacd546b953cad5712e9c9b1410713ea646650e7966cf3ba5524b62f4f18c4e"} Sep 08 04:00:29.573365 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:29.573338 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" event={"ID":"b11bc1d1-6ddb-4e21-8a0f-239e7407f358","Type":"ContainerStarted","Data":"a49510636176ba81801b6f76f3d8363d6ffc622118aa56150fff3ff447026b2b"} Sep 08 04:00:29.574550 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:29.574528 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-drx9g" event={"ID":"f1262da6-55b5-49dd-8b8b-187c86dc46f6","Type":"ContainerStarted","Data":"b4924ac2795deb76c224d71be9dfd082cbdb803649b32431f1757fca4ff08bb3"} Sep 08 04:00:29.575751 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:29.575724 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qcbd8" event={"ID":"6698d58b-300e-4f04-a773-1756fcd6f558","Type":"ContainerStarted","Data":"40109c53b717cea9ef2618ca527b20fc056f6561c883eb8c5b686bc873651b03"} Sep 08 04:00:29.576824 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:29.576805 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-62m8t" event={"ID":"c0f49eb0-c8d2-4383-a286-366557ec8894","Type":"ContainerStarted","Data":"bf0535783cafe7969478e9195c40db9eac95cbbe2a05ad4af4a1402f305ca252"} Sep 08 04:00:29.577856 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:29.577837 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d95fg" event={"ID":"ce867bba-0058-4825-bad0-6962d3ea4890","Type":"ContainerStarted","Data":"d2b1deac807beff19e85a1613b0d1cd191abf1252f93e24cb2a024eccb9a802b"} Sep 08 04:00:29.579199 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:29.579178 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86px5" event={"ID":"b2959761-d748-49e4-8e37-c31e3c0517a8","Type":"ContainerStarted","Data":"6dd87d089f1b986fabb6eb59bac4938f07a28b3b20f64da1d3d80bce08f56da1"} Sep 08 04:00:29.581313 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:29.581290 2574 generic.go:356] "Generic (PLEG): container finished" podID="8da85c75381fab0440e1fd992aacf5e7" containerID="3aff15236e38c45d36e298d618304b2d10d50ecaf0994ff87519f5705e184632" exitCode=0 Sep 08 04:00:29.581394 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:29.581355 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal" event={"ID":"8da85c75381fab0440e1fd992aacf5e7","Type":"ContainerDied","Data":"3aff15236e38c45d36e298d618304b2d10d50ecaf0994ff87519f5705e184632"} Sep 08 04:00:29.582642 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:29.582622 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-84pxf" event={"ID":"fab1c08f-5e53-42c6-a080-e510ae257469","Type":"ContainerStarted","Data":"1b734d0a2f16d50e2673d8d1539e9b67fcca44c31978f37492fbc03f61ae7b1d"} Sep 08 04:00:29.583888 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:29.583860 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" event={"ID":"3d167280-ff71-4e85-9abd-41b51f9358c6","Type":"ContainerStarted","Data":"0d8b7674a116ccf8e2d1fb0145ca53c42f4e5a814fda9b3b686b63d0910b67bd"} Sep 08 04:00:30.056000 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:30.055965 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff12a182-ed89-4513-938b-69971199ad25-metrics-certs\") pod \"network-metrics-daemon-rzwvc\" (UID: \"ff12a182-ed89-4513-938b-69971199ad25\") " pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:00:30.056189 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:30.056164 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 04:00:30.056255 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:30.056245 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff12a182-ed89-4513-938b-69971199ad25-metrics-certs podName:ff12a182-ed89-4513-938b-69971199ad25 nodeName:}" failed. No retries permitted until 2026-09-08 04:00:32.05622257 +0000 UTC m=+5.978683305 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff12a182-ed89-4513-938b-69971199ad25-metrics-certs") pod "network-metrics-daemon-rzwvc" (UID: "ff12a182-ed89-4513-938b-69971199ad25") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 04:00:30.157216 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:30.157171 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-original-pull-secret\") pod \"global-pull-secret-syncer-x4dkl\" (UID: \"c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb\") " pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:30.157405 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:30.157340 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 04:00:30.157451 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:30.157425 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-original-pull-secret podName:c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb nodeName:}" failed. No retries permitted until 2026-09-08 04:00:32.157404061 +0000 UTC m=+6.079864795 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-original-pull-secret") pod "global-pull-secret-syncer-x4dkl" (UID: "c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb") : object "kube-system"/"original-pull-secret" not registered Sep 08 04:00:30.257989 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:30.257958 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tns48\" (UniqueName: \"kubernetes.io/projected/4c61ba14-8d9f-4634-9a0c-aac888042abe-kube-api-access-tns48\") pod \"network-check-target-l6l2z\" (UID: \"4c61ba14-8d9f-4634-9a0c-aac888042abe\") " pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:00:30.258203 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:30.258180 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 04:00:30.258274 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:30.258214 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 04:00:30.258274 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:30.258229 2574 projected.go:196] Error preparing data for projected volume kube-api-access-tns48 for pod openshift-network-diagnostics/network-check-target-l6l2z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 04:00:30.258397 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:30.258298 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c61ba14-8d9f-4634-9a0c-aac888042abe-kube-api-access-tns48 podName:4c61ba14-8d9f-4634-9a0c-aac888042abe nodeName:}" failed. No retries permitted until 2026-09-08 04:00:32.258278663 +0000 UTC m=+6.180739397 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tns48" (UniqueName: "kubernetes.io/projected/4c61ba14-8d9f-4634-9a0c-aac888042abe-kube-api-access-tns48") pod "network-check-target-l6l2z" (UID: "4c61ba14-8d9f-4634-9a0c-aac888042abe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 04:00:30.475417 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:30.475376 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 03:55:28 +0000 UTC" deadline="2028-02-21 02:53:29.399286912 +0000 UTC" Sep 08 04:00:30.475417 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:30.475415 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12742h52m58.923876921s" Sep 08 04:00:30.564796 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:30.564721 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:00:30.564941 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:30.564875 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzwvc" podUID="ff12a182-ed89-4513-938b-69971199ad25" Sep 08 04:00:30.589442 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:30.589409 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal" event={"ID":"8da85c75381fab0440e1fd992aacf5e7","Type":"ContainerStarted","Data":"9def3e8bfc29857f9fde82989fe4e443e5da2fffd10d006c33553962d476091b"} Sep 08 04:00:30.592123 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:30.592095 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-37.ec2.internal" event={"ID":"480faa79f347b524a5131c7d80428e3a","Type":"ContainerStarted","Data":"9058a14bdae54fc5bfced60460dbabc7635b985c29c7f713100c33cd56634ff5"} Sep 08 04:00:30.633555 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:30.633498 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal" podStartSLOduration=2.633478608 podStartE2EDuration="2.633478608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 04:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:00:30.613768687 +0000 UTC m=+4.536229450" watchObservedRunningTime="2026-09-08 04:00:30.633478608 +0000 UTC m=+4.555939353" Sep 08 04:00:31.562209 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:31.561507 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:00:31.562209 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:31.561668 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6l2z" podUID="4c61ba14-8d9f-4634-9a0c-aac888042abe" Sep 08 04:00:31.562209 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:31.562061 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:31.562209 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:31.562166 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4dkl" podUID="c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb" Sep 08 04:00:32.075405 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:32.075078 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff12a182-ed89-4513-938b-69971199ad25-metrics-certs\") pod \"network-metrics-daemon-rzwvc\" (UID: \"ff12a182-ed89-4513-938b-69971199ad25\") " pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:00:32.075405 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:32.075328 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 04:00:32.075405 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:32.075404 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff12a182-ed89-4513-938b-69971199ad25-metrics-certs podName:ff12a182-ed89-4513-938b-69971199ad25 nodeName:}" failed. No retries permitted until 2026-09-08 04:00:36.075382283 +0000 UTC m=+9.997843022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff12a182-ed89-4513-938b-69971199ad25-metrics-certs") pod "network-metrics-daemon-rzwvc" (UID: "ff12a182-ed89-4513-938b-69971199ad25") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 04:00:32.176915 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:32.176877 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-original-pull-secret\") pod \"global-pull-secret-syncer-x4dkl\" (UID: \"c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb\") " pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:32.177146 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:32.177124 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 04:00:32.177237 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:32.177205 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-original-pull-secret podName:c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb nodeName:}" failed. No retries permitted until 2026-09-08 04:00:36.177183731 +0000 UTC m=+10.099644450 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-original-pull-secret") pod "global-pull-secret-syncer-x4dkl" (UID: "c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb") : object "kube-system"/"original-pull-secret" not registered Sep 08 04:00:32.278102 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:32.278034 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tns48\" (UniqueName: \"kubernetes.io/projected/4c61ba14-8d9f-4634-9a0c-aac888042abe-kube-api-access-tns48\") pod \"network-check-target-l6l2z\" (UID: \"4c61ba14-8d9f-4634-9a0c-aac888042abe\") " pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:00:32.278274 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:32.278200 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 04:00:32.278274 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:32.278223 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 04:00:32.278274 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:32.278235 2574 projected.go:196] Error preparing data for projected volume kube-api-access-tns48 for pod openshift-network-diagnostics/network-check-target-l6l2z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 04:00:32.278425 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:32.278292 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c61ba14-8d9f-4634-9a0c-aac888042abe-kube-api-access-tns48 podName:4c61ba14-8d9f-4634-9a0c-aac888042abe nodeName:}" failed. No retries permitted until 2026-09-08 04:00:36.278273976 +0000 UTC m=+10.200734696 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tns48" (UniqueName: "kubernetes.io/projected/4c61ba14-8d9f-4634-9a0c-aac888042abe-kube-api-access-tns48") pod "network-check-target-l6l2z" (UID: "4c61ba14-8d9f-4634-9a0c-aac888042abe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 04:00:32.563181 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:32.562783 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:00:32.563181 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:32.562936 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzwvc" podUID="ff12a182-ed89-4513-938b-69971199ad25" Sep 08 04:00:33.563043 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:33.562355 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:33.563043 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:33.562492 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4dkl" podUID="c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb" Sep 08 04:00:33.563043 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:33.562886 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:00:33.563043 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:33.562987 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6l2z" podUID="4c61ba14-8d9f-4634-9a0c-aac888042abe" Sep 08 04:00:34.566104 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:34.565700 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:00:34.566104 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:34.565826 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzwvc" podUID="ff12a182-ed89-4513-938b-69971199ad25" Sep 08 04:00:35.562127 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:35.561472 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:00:35.562127 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:35.561619 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6l2z" podUID="4c61ba14-8d9f-4634-9a0c-aac888042abe" Sep 08 04:00:35.562127 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:35.561989 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:35.562127 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:35.562083 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4dkl" podUID="c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb" Sep 08 04:00:36.112514 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:36.112459 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff12a182-ed89-4513-938b-69971199ad25-metrics-certs\") pod \"network-metrics-daemon-rzwvc\" (UID: \"ff12a182-ed89-4513-938b-69971199ad25\") " pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:00:36.112946 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:36.112610 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 04:00:36.112946 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:36.112672 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff12a182-ed89-4513-938b-69971199ad25-metrics-certs podName:ff12a182-ed89-4513-938b-69971199ad25 nodeName:}" failed. No retries permitted until 2026-09-08 04:00:44.11265418 +0000 UTC m=+18.035114913 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff12a182-ed89-4513-938b-69971199ad25-metrics-certs") pod "network-metrics-daemon-rzwvc" (UID: "ff12a182-ed89-4513-938b-69971199ad25") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 04:00:36.213399 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:36.212846 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-original-pull-secret\") pod \"global-pull-secret-syncer-x4dkl\" (UID: \"c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb\") " pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:36.213399 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:36.212980 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 04:00:36.213399 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:36.213037 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-original-pull-secret podName:c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb nodeName:}" failed. No retries permitted until 2026-09-08 04:00:44.21302063 +0000 UTC m=+18.135481351 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-original-pull-secret") pod "global-pull-secret-syncer-x4dkl" (UID: "c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb") : object "kube-system"/"original-pull-secret" not registered Sep 08 04:00:36.313641 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:36.313607 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tns48\" (UniqueName: \"kubernetes.io/projected/4c61ba14-8d9f-4634-9a0c-aac888042abe-kube-api-access-tns48\") pod \"network-check-target-l6l2z\" (UID: \"4c61ba14-8d9f-4634-9a0c-aac888042abe\") " pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:00:36.313866 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:36.313819 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 04:00:36.313866 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:36.313846 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 04:00:36.313866 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:36.313858 2574 projected.go:196] Error preparing data for projected volume kube-api-access-tns48 for pod openshift-network-diagnostics/network-check-target-l6l2z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 04:00:36.314015 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:36.313913 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c61ba14-8d9f-4634-9a0c-aac888042abe-kube-api-access-tns48 podName:4c61ba14-8d9f-4634-9a0c-aac888042abe nodeName:}" failed. No retries permitted until 2026-09-08 04:00:44.313899376 +0000 UTC m=+18.236360097 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tns48" (UniqueName: "kubernetes.io/projected/4c61ba14-8d9f-4634-9a0c-aac888042abe-kube-api-access-tns48") pod "network-check-target-l6l2z" (UID: "4c61ba14-8d9f-4634-9a0c-aac888042abe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 04:00:36.562999 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:36.562965 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:00:36.563170 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:36.563086 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzwvc" podUID="ff12a182-ed89-4513-938b-69971199ad25" Sep 08 04:00:37.562248 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:37.561788 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:37.562248 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:37.561914 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4dkl" podUID="c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb" Sep 08 04:00:37.562248 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:37.562107 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:00:37.562248 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:37.562213 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6l2z" podUID="4c61ba14-8d9f-4634-9a0c-aac888042abe" Sep 08 04:00:38.563292 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:38.563261 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:00:38.563744 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:38.563396 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzwvc" podUID="ff12a182-ed89-4513-938b-69971199ad25" Sep 08 04:00:39.561867 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:39.561839 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:39.562005 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:39.561845 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:00:39.562048 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:39.562006 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6l2z" podUID="4c61ba14-8d9f-4634-9a0c-aac888042abe" Sep 08 04:00:39.562048 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:39.561930 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4dkl" podUID="c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb" Sep 08 04:00:40.564774 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:40.564744 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:00:40.565117 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:40.564848 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzwvc" podUID="ff12a182-ed89-4513-938b-69971199ad25" Sep 08 04:00:41.561794 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:41.561762 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:41.561980 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:41.561881 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4dkl" podUID="c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb" Sep 08 04:00:41.561980 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:41.561934 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:00:41.562089 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:41.562040 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6l2z" podUID="4c61ba14-8d9f-4634-9a0c-aac888042abe" Sep 08 04:00:42.561904 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:42.561875 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:00:42.562363 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:42.562003 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzwvc" podUID="ff12a182-ed89-4513-938b-69971199ad25" Sep 08 04:00:43.562420 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:43.562390 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:00:43.562857 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:43.562390 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:43.562857 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:43.562486 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6l2z" podUID="4c61ba14-8d9f-4634-9a0c-aac888042abe" Sep 08 04:00:43.562857 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:43.562609 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4dkl" podUID="c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb" Sep 08 04:00:44.179053 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:44.179015 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff12a182-ed89-4513-938b-69971199ad25-metrics-certs\") pod \"network-metrics-daemon-rzwvc\" (UID: \"ff12a182-ed89-4513-938b-69971199ad25\") " pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:00:44.179236 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:44.179165 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 04:00:44.179236 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:44.179229 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff12a182-ed89-4513-938b-69971199ad25-metrics-certs podName:ff12a182-ed89-4513-938b-69971199ad25 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:00.179210015 +0000 UTC m=+34.101670745 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff12a182-ed89-4513-938b-69971199ad25-metrics-certs") pod "network-metrics-daemon-rzwvc" (UID: "ff12a182-ed89-4513-938b-69971199ad25") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 04:00:44.279525 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:44.279483 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-original-pull-secret\") pod \"global-pull-secret-syncer-x4dkl\" (UID: \"c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb\") " pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:44.279679 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:44.279631 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 04:00:44.279730 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:44.279691 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-original-pull-secret podName:c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb nodeName:}" failed. No retries permitted until 2026-09-08 04:01:00.279672586 +0000 UTC m=+34.202133303 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-original-pull-secret") pod "global-pull-secret-syncer-x4dkl" (UID: "c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb") : object "kube-system"/"original-pull-secret" not registered Sep 08 04:00:44.381267 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:44.381226 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tns48\" (UniqueName: \"kubernetes.io/projected/4c61ba14-8d9f-4634-9a0c-aac888042abe-kube-api-access-tns48\") pod \"network-check-target-l6l2z\" (UID: \"4c61ba14-8d9f-4634-9a0c-aac888042abe\") " pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:00:44.381811 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:44.381452 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 04:00:44.381811 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:44.381475 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 04:00:44.381811 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:44.381559 2574 projected.go:196] Error preparing data for projected volume kube-api-access-tns48 for pod openshift-network-diagnostics/network-check-target-l6l2z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 04:00:44.381811 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:44.381638 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c61ba14-8d9f-4634-9a0c-aac888042abe-kube-api-access-tns48 podName:4c61ba14-8d9f-4634-9a0c-aac888042abe nodeName:}" failed. No retries permitted until 2026-09-08 04:01:00.381610595 +0000 UTC m=+34.304071333 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tns48" (UniqueName: "kubernetes.io/projected/4c61ba14-8d9f-4634-9a0c-aac888042abe-kube-api-access-tns48") pod "network-check-target-l6l2z" (UID: "4c61ba14-8d9f-4634-9a0c-aac888042abe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 04:00:44.565420 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:44.565397 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:00:44.565837 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:44.565500 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzwvc" podUID="ff12a182-ed89-4513-938b-69971199ad25" Sep 08 04:00:45.561481 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:45.561452 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:45.561662 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:45.561452 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:00:45.561662 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:45.561557 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4dkl" podUID="c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb" Sep 08 04:00:45.561662 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:45.561649 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6l2z" podUID="4c61ba14-8d9f-4634-9a0c-aac888042abe" Sep 08 04:00:46.562739 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:46.562708 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:00:46.563073 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:46.562796 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzwvc" podUID="ff12a182-ed89-4513-938b-69971199ad25" Sep 08 04:00:47.561795 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:47.561631 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:00:47.561883 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:47.561867 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6l2z" podUID="4c61ba14-8d9f-4634-9a0c-aac888042abe" Sep 08 04:00:47.561933 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:47.561632 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:47.561933 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:47.561929 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4dkl" podUID="c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb" Sep 08 04:00:47.634972 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:47.634948 2574 generic.go:356] "Generic (PLEG): container finished" podID="8ac44940-1e7f-48c1-a9db-8ac81a574f45" containerID="ab60f6d675393d287fb901c9a50d36e02b2ab2d2eb37600d32367602bf0b521f" exitCode=0 Sep 08 04:00:47.635736 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:47.635028 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrhb8" event={"ID":"8ac44940-1e7f-48c1-a9db-8ac81a574f45","Type":"ContainerDied","Data":"ab60f6d675393d287fb901c9a50d36e02b2ab2d2eb37600d32367602bf0b521f"} Sep 08 04:00:47.636747 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:47.636723 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" event={"ID":"b11bc1d1-6ddb-4e21-8a0f-239e7407f358","Type":"ContainerStarted","Data":"aafa788d0e53d9e40e955678cfca0dc3ef853e1f62b1e7a5fab54031e4d24ee5"} Sep 08 04:00:47.638390 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:47.638367 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-drx9g" event={"ID":"f1262da6-55b5-49dd-8b8b-187c86dc46f6","Type":"ContainerStarted","Data":"d53c06e18c8c065f3ea038deb8cc729f8de0b1e46e99c83288cdc68d04af709c"} Sep 08 04:00:47.640024 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:47.640001 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qcbd8" event={"ID":"6698d58b-300e-4f04-a773-1756fcd6f558","Type":"ContainerStarted","Data":"264610d432e7393810e641298d8ab33d8e044f9d450dcc8ae05341053582f60c"} Sep 08 04:00:47.641632 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:47.641613 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d95fg" event={"ID":"ce867bba-0058-4825-bad0-6962d3ea4890","Type":"ContainerStarted","Data":"8bed70987b6ec44f1353ef5629a37412b5189bd00dd8f045aa729657f53a623f"} Sep 08 04:00:47.644866 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:47.644846 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86px5" event={"ID":"b2959761-d748-49e4-8e37-c31e3c0517a8","Type":"ContainerStarted","Data":"9a23b8b36adf3ce7283af2e80435dfdc0d5c8fdb03536623f651ccd0239f5999"} Sep 08 04:00:47.644948 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:47.644918 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86px5" event={"ID":"b2959761-d748-49e4-8e37-c31e3c0517a8","Type":"ContainerStarted","Data":"ad60c12f465087934f32d8089740f7ac1fc4524b300ef3a210a1ea316f4b18a3"} Sep 08 04:00:47.645100 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:47.644969 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86px5" event={"ID":"b2959761-d748-49e4-8e37-c31e3c0517a8","Type":"ContainerStarted","Data":"74fc8ddd15f7608b600e6d64c047a542b94ac81da1118669d01f1cb86f614607"} Sep 08 04:00:47.645100 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:47.645017 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86px5" event={"ID":"b2959761-d748-49e4-8e37-c31e3c0517a8","Type":"ContainerStarted","Data":"c0ec47b903b6ca8c7b9cf0c4f099d124f5297e286b7a8ec64a3fd44339b848d8"} Sep 08 04:00:47.645100 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:47.645055 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86px5" event={"ID":"b2959761-d748-49e4-8e37-c31e3c0517a8","Type":"ContainerStarted","Data":"e2927fb6adc6f25d4a73fb4def12c1423e60472285cd2943bffb337deff58b39"} Sep 08 04:00:47.645100 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:47.645094 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86px5" event={"ID":"b2959761-d748-49e4-8e37-c31e3c0517a8","Type":"ContainerStarted","Data":"b27daf3549b618c410029b5391ede5ea68f0ad45e5eba5500b611438647a5192"} Sep 08 04:00:47.646144 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:47.646116 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-84pxf" event={"ID":"fab1c08f-5e53-42c6-a080-e510ae257469","Type":"ContainerStarted","Data":"8af2063cd17c1f4c5af86cdbd2379d4968b1100668be5db33643a6f2f80a3195"} Sep 08 04:00:47.647614 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:47.647593 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" event={"ID":"3d167280-ff71-4e85-9abd-41b51f9358c6","Type":"ContainerStarted","Data":"8b54b15dfd49b435e86fae5eda29c8533649dcd17f66e4c23c47b81439f15ded"} Sep 08 04:00:47.654798 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:47.654762 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-37.ec2.internal" podStartSLOduration=19.654751498 podStartE2EDuration="19.654751498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 04:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:00:30.634680239 +0000 UTC m=+4.557141004" watchObservedRunningTime="2026-09-08 04:00:47.654751498 +0000 UTC m=+21.577212237" Sep 08 04:00:47.670056 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:47.670020 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hw7xq" podStartSLOduration=2.775026718 podStartE2EDuration="20.670008275s" totalImagesPullingTime="17.894981557s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 04:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:00:47.669891339 +0000 UTC m=+21.592352109" watchObservedRunningTime="2026-09-08 04:00:47.670008275 +0000 UTC m=+21.592469014" Sep 08 04:00:47.684264 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:47.684231 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-drx9g" podStartSLOduration=6.7879198800000005 podStartE2EDuration="20.684222499s" totalImagesPullingTime="13.896302619s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 04:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:00:47.684018307 +0000 UTC m=+21.606479071" watchObservedRunningTime="2026-09-08 04:00:47.684222499 +0000 UTC m=+21.606683237" Sep 08 04:00:47.697692 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:47.697653 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-84pxf" podStartSLOduration=2.846231618 podStartE2EDuration="20.69763738s" totalImagesPullingTime="17.851405762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:00:47.697484761 +0000 UTC m=+21.619945523" watchObservedRunningTime="2026-09-08 04:00:47.69763738 +0000 UTC m=+21.620098118" Sep 08 04:00:47.715318 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:47.715281 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-d95fg" podStartSLOduration=2.789247316 podStartE2EDuration="20.715268468s" totalImagesPullingTime="17.926021152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 04:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:00:47.714887779 +0000 UTC m=+21.637348547" watchObservedRunningTime="2026-09-08 04:00:47.715268468 +0000 UTC m=+21.637729205" Sep 08 04:00:47.729951 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:47.729872 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qcbd8" podStartSLOduration=10.376103843 podStartE2EDuration="20.729855084s" totalImagesPullingTime="10.353751241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 04:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:00:47.729034609 +0000 UTC m=+21.651495373" watchObservedRunningTime="2026-09-08 04:00:47.729855084 +0000 UTC m=+21.652315822" Sep 08 04:00:48.139969 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:48.139798 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 08 04:00:48.483815 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:48.483717 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T04:00:48.139955018Z","UUID":"fd056a39-f93b-4100-b92d-32cfbc793301","Handler":null,"Name":"","Endpoint":""} Sep 08 04:00:48.485372 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:48.485352 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 08 04:00:48.485476 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:48.485384 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 08 04:00:48.561935 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:48.561905 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:00:48.562086 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:48.562023 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzwvc" podUID="ff12a182-ed89-4513-938b-69971199ad25" Sep 08 04:00:48.652109 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:48.652043 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" event={"ID":"b11bc1d1-6ddb-4e21-8a0f-239e7407f358","Type":"ContainerStarted","Data":"8914d9ab9c9937951298efd0c18a27d94208b2be455f4e65958212a6e9b877d7"} Sep 08 04:00:48.653616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:48.653574 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-62m8t" event={"ID":"c0f49eb0-c8d2-4383-a286-366557ec8894","Type":"ContainerStarted","Data":"07e27da3a33fae474f2bf48f586311c9089eb88a74af76267136cd6e588a9308"} Sep 08 04:00:48.670359 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:48.670318 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-62m8t" podStartSLOduration=11.313588394 podStartE2EDuration="21.670306629s" totalImagesPullingTime="10.356718235s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 04:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:00:48.670117045 +0000 UTC m=+22.592577815" watchObservedRunningTime="2026-09-08 04:00:48.670306629 +0000 UTC m=+22.592767368" Sep 08 04:00:49.562348 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:49.562315 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:49.562560 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:49.562353 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:00:49.562560 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:49.562430 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4dkl" podUID="c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb" Sep 08 04:00:49.562673 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:49.562576 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6l2z" podUID="4c61ba14-8d9f-4634-9a0c-aac888042abe" Sep 08 04:00:49.658032 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:49.657999 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" event={"ID":"b11bc1d1-6ddb-4e21-8a0f-239e7407f358","Type":"ContainerStarted","Data":"b6bcbbc161cba0c44f63da98ed87783fe6cb0042e97c65d57114c15497ad285b"} Sep 08 04:00:49.675035 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:49.674997 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5xtt" podStartSLOduration=2.947207184 podStartE2EDuration="22.67498392s" totalImagesPullingTime="19.727776736s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 04:00:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:00:49.674492277 +0000 UTC m=+23.596953034" watchObservedRunningTime="2026-09-08 04:00:49.67498392 +0000 UTC m=+23.597444658" Sep 08 04:00:50.562602 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:50.562572 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:00:50.562883 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:50.562701 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzwvc" podUID="ff12a182-ed89-4513-938b-69971199ad25" Sep 08 04:00:50.664214 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:50.664178 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86px5" event={"ID":"b2959761-d748-49e4-8e37-c31e3c0517a8","Type":"ContainerStarted","Data":"65a5a7a0fd4e62522839c0c9457497dfc40dd113e4b05f49b9bb56feb7304134"} Sep 08 04:00:51.562012 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:51.561983 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:51.562186 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:51.561983 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:00:51.562186 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:51.562088 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4dkl" podUID="c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb" Sep 08 04:00:51.562284 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:51.562188 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6l2z" podUID="4c61ba14-8d9f-4634-9a0c-aac888042abe" Sep 08 04:00:52.491814 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:52.491491 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-84pxf" Sep 08 04:00:52.492679 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:52.492080 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-84pxf" Sep 08 04:00:52.561609 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:52.561584 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:00:52.561758 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:52.561680 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzwvc" podUID="ff12a182-ed89-4513-938b-69971199ad25" Sep 08 04:00:52.669910 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:52.669881 2574 generic.go:356] "Generic (PLEG): container finished" podID="8ac44940-1e7f-48c1-a9db-8ac81a574f45" containerID="30ed691b39d782087dc0985ac74dd3d68b196b0243d4c49106e9de828330b622" exitCode=0 Sep 08 04:00:52.670071 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:52.669935 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrhb8" event={"ID":"8ac44940-1e7f-48c1-a9db-8ac81a574f45","Type":"ContainerDied","Data":"30ed691b39d782087dc0985ac74dd3d68b196b0243d4c49106e9de828330b622"} Sep 08 04:00:52.673615 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:52.673590 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86px5" event={"ID":"b2959761-d748-49e4-8e37-c31e3c0517a8","Type":"ContainerStarted","Data":"7d0a8cb2532c7c3354908de61d91c18f656d36d8be4941aae9e95868f2eec00a"} Sep 08 04:00:52.673913 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:52.673889 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-84pxf" Sep 08 04:00:52.674265 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:52.674249 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-84pxf" Sep 08 04:00:52.720885 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:52.720825 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-86px5" podStartSLOduration=7.574055035 podStartE2EDuration="25.72080748s" totalImagesPullingTime="18.146752445s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 04:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:00:52.719848765 +0000 UTC m=+26.642309525" watchObservedRunningTime="2026-09-08 04:00:52.72080748 +0000 UTC m=+26.643268219" Sep 08 04:00:53.562210 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:53.562182 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:53.562633 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:53.562194 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:00:53.562633 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:53.562318 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4dkl" podUID="c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb" Sep 08 04:00:53.562633 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:53.562342 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6l2z" podUID="4c61ba14-8d9f-4634-9a0c-aac888042abe" Sep 08 04:00:53.677413 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:53.677382 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrhb8" event={"ID":"8ac44940-1e7f-48c1-a9db-8ac81a574f45","Type":"ContainerStarted","Data":"0a1f2235f698b5aafed67b5e6bcd4af79e1b22d66f07a6ebb18a8ea0c803e55c"} Sep 08 04:00:53.677521 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:53.677452 2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 08 04:00:53.677829 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:53.677814 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:53.677895 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:53.677887 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:53.695567 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:53.695546 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:53.708587 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:53.708569 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:53.741546 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:53.741521 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rzwvc"] Sep 08 04:00:53.741659 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:53.741643 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:00:53.741748 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:53.741733 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzwvc" podUID="ff12a182-ed89-4513-938b-69971199ad25" Sep 08 04:00:53.744672 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:53.744649 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x4dkl"] Sep 08 04:00:53.744778 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:53.744714 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:53.744833 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:53.744790 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4dkl" podUID="c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb" Sep 08 04:00:53.745439 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:53.745418 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l6l2z"] Sep 08 04:00:53.745525 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:53.745493 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:00:53.745615 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:53.745592 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6l2z" podUID="4c61ba14-8d9f-4634-9a0c-aac888042abe" Sep 08 04:00:54.249042 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:54.249008 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:00:54.681211 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:54.681124 2574 generic.go:356] "Generic (PLEG): container finished" podID="8ac44940-1e7f-48c1-a9db-8ac81a574f45" containerID="0a1f2235f698b5aafed67b5e6bcd4af79e1b22d66f07a6ebb18a8ea0c803e55c" exitCode=0 Sep 08 04:00:54.681687 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:54.681206 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrhb8" event={"ID":"8ac44940-1e7f-48c1-a9db-8ac81a574f45","Type":"ContainerDied","Data":"0a1f2235f698b5aafed67b5e6bcd4af79e1b22d66f07a6ebb18a8ea0c803e55c"} Sep 08 04:00:55.099963 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:55.099926 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:00:55.193682 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:55.193650 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:00:55.561559 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:55.561531 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:00:55.561729 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:55.561565 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:00:55.561729 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:55.561538 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:55.561729 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:55.561635 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6l2z" podUID="4c61ba14-8d9f-4634-9a0c-aac888042abe" Sep 08 04:00:55.561729 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:55.561714 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4dkl" podUID="c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb" Sep 08 04:00:55.561855 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:55.561795 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzwvc" podUID="ff12a182-ed89-4513-938b-69971199ad25" Sep 08 04:00:55.685726 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:55.685697 2574 generic.go:356] "Generic (PLEG): container finished" podID="8ac44940-1e7f-48c1-a9db-8ac81a574f45" containerID="64e498d71792891070a5a3b1877ea29f798020dd7ef02d18cfb014b0275c58c5" exitCode=0 Sep 08 04:00:55.686091 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:55.685781 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrhb8" event={"ID":"8ac44940-1e7f-48c1-a9db-8ac81a574f45","Type":"ContainerDied","Data":"64e498d71792891070a5a3b1877ea29f798020dd7ef02d18cfb014b0275c58c5"} Sep 08 04:00:57.562417 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:57.562217 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:57.562880 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:57.562217 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:00:57.562880 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:57.562521 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4dkl" podUID="c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb" Sep 08 04:00:57.562880 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:57.562618 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6l2z" podUID="4c61ba14-8d9f-4634-9a0c-aac888042abe" Sep 08 04:00:57.562880 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:57.562222 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:00:57.562880 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:57.562724 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzwvc" podUID="ff12a182-ed89-4513-938b-69971199ad25" Sep 08 04:00:59.354178 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:59.354140 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:00:59.455216 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:59.455186 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:00:59.561760 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:59.561728 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:00:59.561916 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:59.561728 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:00:59.561916 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:59.561858 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6l2z" podUID="4c61ba14-8d9f-4634-9a0c-aac888042abe" Sep 08 04:00:59.562010 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:59.561728 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:00:59.562309 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:59.562276 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzwvc" podUID="ff12a182-ed89-4513-938b-69971199ad25" Sep 08 04:00:59.562453 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:00:59.562429 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4dkl" podUID="c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb" Sep 08 04:00:59.875996 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:59.875929 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeReady" Sep 08 04:00:59.876112 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:59.876059 2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 08 04:00:59.924050 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:59.923935 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-b8vzf"] Sep 08 04:00:59.946409 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:59.946388 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pv6qv"] Sep 08 04:00:59.946554 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:59.946535 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b8vzf" Sep 08 04:00:59.949059 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:59.949036 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 08 04:00:59.949200 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:59.949105 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 08 04:00:59.949291 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:59.949272 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-26sdq\"" Sep 08 04:00:59.958436 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:59.958417 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b8vzf"] Sep 08 04:00:59.958560 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:59.958446 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pv6qv"] Sep 08 04:00:59.958560 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:59.958526 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pv6qv" Sep 08 04:00:59.960931 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:59.960879 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 08 04:00:59.961044 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:59.961002 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 08 04:00:59.961136 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:59.961116 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9nxct\"" Sep 08 04:00:59.961198 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:00:59.961181 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 08 04:01:00.068841 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:00.068813 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:00.096791 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:00.096763 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e2eec1f-b2a7-43d5-a9e7-f15eafb23937-cert\") pod \"ingress-canary-pv6qv\" (UID: \"3e2eec1f-b2a7-43d5-a9e7-f15eafb23937\") " pod="openshift-ingress-canary/ingress-canary-pv6qv" Sep 08 04:01:00.096901 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:00.096803 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qszh5\" (UniqueName: \"kubernetes.io/projected/57a0643d-28d1-4d11-bb40-1de33627ca6d-kube-api-access-qszh5\") pod \"dns-default-b8vzf\" (UID: \"57a0643d-28d1-4d11-bb40-1de33627ca6d\") " pod="openshift-dns/dns-default-b8vzf" Sep 08 04:01:00.096901 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:00.096829 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57a0643d-28d1-4d11-bb40-1de33627ca6d-config-volume\") pod \"dns-default-b8vzf\" (UID: \"57a0643d-28d1-4d11-bb40-1de33627ca6d\") " pod="openshift-dns/dns-default-b8vzf" Sep 08 04:01:00.096987 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:00.096924 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flv9z\" (UniqueName: \"kubernetes.io/projected/3e2eec1f-b2a7-43d5-a9e7-f15eafb23937-kube-api-access-flv9z\") pod \"ingress-canary-pv6qv\" (UID: \"3e2eec1f-b2a7-43d5-a9e7-f15eafb23937\") " pod="openshift-ingress-canary/ingress-canary-pv6qv" Sep 08 04:01:00.096987 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:00.096962 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57a0643d-28d1-4d11-bb40-1de33627ca6d-tmp-dir\") pod \"dns-default-b8vzf\" (UID: \"57a0643d-28d1-4d11-bb40-1de33627ca6d\") " pod="openshift-dns/dns-default-b8vzf" Sep 08 04:01:00.097066 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:00.097002 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57a0643d-28d1-4d11-bb40-1de33627ca6d-metrics-tls\") pod \"dns-default-b8vzf\" (UID: \"57a0643d-28d1-4d11-bb40-1de33627ca6d\") " pod="openshift-dns/dns-default-b8vzf" Sep 08 04:01:00.131761 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:00.131705 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:00.198276 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:00.198251 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff12a182-ed89-4513-938b-69971199ad25-metrics-certs\") pod \"network-metrics-daemon-rzwvc\" (UID: \"ff12a182-ed89-4513-938b-69971199ad25\") " pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:01:00.198399 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:00.198291 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57a0643d-28d1-4d11-bb40-1de33627ca6d-metrics-tls\") pod \"dns-default-b8vzf\" (UID: \"57a0643d-28d1-4d11-bb40-1de33627ca6d\") " pod="openshift-dns/dns-default-b8vzf" Sep 08 04:01:00.198399 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:00.198336 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e2eec1f-b2a7-43d5-a9e7-f15eafb23937-cert\") pod \"ingress-canary-pv6qv\" (UID: \"3e2eec1f-b2a7-43d5-a9e7-f15eafb23937\") " pod="openshift-ingress-canary/ingress-canary-pv6qv" Sep 08 04:01:00.198399 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:00.198385 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qszh5\" (UniqueName: \"kubernetes.io/projected/57a0643d-28d1-4d11-bb40-1de33627ca6d-kube-api-access-qszh5\") pod \"dns-default-b8vzf\" (UID: \"57a0643d-28d1-4d11-bb40-1de33627ca6d\") " pod="openshift-dns/dns-default-b8vzf" Sep 08 04:01:00.198571 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:00.198402 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 04:01:00.198571 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:00.198411 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57a0643d-28d1-4d11-bb40-1de33627ca6d-config-volume\") pod \"dns-default-b8vzf\" (UID: \"57a0643d-28d1-4d11-bb40-1de33627ca6d\") " pod="openshift-dns/dns-default-b8vzf" Sep 08 04:01:00.198571 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:00.198452 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff12a182-ed89-4513-938b-69971199ad25-metrics-certs podName:ff12a182-ed89-4513-938b-69971199ad25 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:32.198438425 +0000 UTC m=+66.120899146 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff12a182-ed89-4513-938b-69971199ad25-metrics-certs") pod "network-metrics-daemon-rzwvc" (UID: "ff12a182-ed89-4513-938b-69971199ad25") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 04:01:00.198571 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:00.198485 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flv9z\" (UniqueName: \"kubernetes.io/projected/3e2eec1f-b2a7-43d5-a9e7-f15eafb23937-kube-api-access-flv9z\") pod \"ingress-canary-pv6qv\" (UID: \"3e2eec1f-b2a7-43d5-a9e7-f15eafb23937\") " pod="openshift-ingress-canary/ingress-canary-pv6qv" Sep 08 04:01:00.198571 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:00.198522 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57a0643d-28d1-4d11-bb40-1de33627ca6d-tmp-dir\") pod \"dns-default-b8vzf\" (UID: \"57a0643d-28d1-4d11-bb40-1de33627ca6d\") " pod="openshift-dns/dns-default-b8vzf" Sep 08 04:01:00.198849 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:00.198820 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57a0643d-28d1-4d11-bb40-1de33627ca6d-tmp-dir\") pod \"dns-default-b8vzf\" (UID: \"57a0643d-28d1-4d11-bb40-1de33627ca6d\") " pod="openshift-dns/dns-default-b8vzf" Sep 08 04:01:00.199011 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:00.198989 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57a0643d-28d1-4d11-bb40-1de33627ca6d-config-volume\") pod \"dns-default-b8vzf\" (UID: \"57a0643d-28d1-4d11-bb40-1de33627ca6d\") " pod="openshift-dns/dns-default-b8vzf" Sep 08 04:01:00.199219 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:00.199200 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 04:01:00.199310 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:00.199249 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57a0643d-28d1-4d11-bb40-1de33627ca6d-metrics-tls podName:57a0643d-28d1-4d11-bb40-1de33627ca6d nodeName:}" failed. No retries permitted until 2026-09-08 04:01:00.699233424 +0000 UTC m=+34.621694145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57a0643d-28d1-4d11-bb40-1de33627ca6d-metrics-tls") pod "dns-default-b8vzf" (UID: "57a0643d-28d1-4d11-bb40-1de33627ca6d") : secret "dns-default-metrics-tls" not found Sep 08 04:01:00.204053 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:00.204033 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e2eec1f-b2a7-43d5-a9e7-f15eafb23937-cert\") pod \"ingress-canary-pv6qv\" (UID: \"3e2eec1f-b2a7-43d5-a9e7-f15eafb23937\") " pod="openshift-ingress-canary/ingress-canary-pv6qv" Sep 08 04:01:00.207778 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:00.207757 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qszh5\" (UniqueName: \"kubernetes.io/projected/57a0643d-28d1-4d11-bb40-1de33627ca6d-kube-api-access-qszh5\") pod \"dns-default-b8vzf\" (UID: \"57a0643d-28d1-4d11-bb40-1de33627ca6d\") " pod="openshift-dns/dns-default-b8vzf" Sep 08 04:01:00.208358 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:00.208303 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flv9z\" (UniqueName: \"kubernetes.io/projected/3e2eec1f-b2a7-43d5-a9e7-f15eafb23937-kube-api-access-flv9z\") pod \"ingress-canary-pv6qv\" (UID: \"3e2eec1f-b2a7-43d5-a9e7-f15eafb23937\") " pod="openshift-ingress-canary/ingress-canary-pv6qv" Sep 08 04:01:00.267434 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:00.267410 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pv6qv" Sep 08 04:01:00.299716 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:00.299694 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-original-pull-secret\") pod \"global-pull-secret-syncer-x4dkl\" (UID: \"c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb\") " pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:01:00.299835 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:00.299820 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 04:01:00.299893 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:00.299876 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-original-pull-secret podName:c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb nodeName:}" failed. No retries permitted until 2026-09-08 04:01:32.299858762 +0000 UTC m=+66.222319511 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-original-pull-secret") pod "global-pull-secret-syncer-x4dkl" (UID: "c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb") : object "kube-system"/"original-pull-secret" not registered Sep 08 04:01:00.400104 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:00.400042 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tns48\" (UniqueName: \"kubernetes.io/projected/4c61ba14-8d9f-4634-9a0c-aac888042abe-kube-api-access-tns48\") pod \"network-check-target-l6l2z\" (UID: \"4c61ba14-8d9f-4634-9a0c-aac888042abe\") " pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:01:00.400546 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:00.400175 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 04:01:00.400546 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:00.400192 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 04:01:00.400546 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:00.400204 2574 projected.go:196] Error preparing data for projected volume kube-api-access-tns48 for pod openshift-network-diagnostics/network-check-target-l6l2z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 04:01:00.400546 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:00.400257 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c61ba14-8d9f-4634-9a0c-aac888042abe-kube-api-access-tns48 podName:4c61ba14-8d9f-4634-9a0c-aac888042abe nodeName:}" failed. No retries permitted until 2026-09-08 04:01:32.400244053 +0000 UTC m=+66.322704770 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tns48" (UniqueName: "kubernetes.io/projected/4c61ba14-8d9f-4634-9a0c-aac888042abe-kube-api-access-tns48") pod "network-check-target-l6l2z" (UID: "4c61ba14-8d9f-4634-9a0c-aac888042abe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 04:01:00.702858 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:00.702801 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57a0643d-28d1-4d11-bb40-1de33627ca6d-metrics-tls\") pod \"dns-default-b8vzf\" (UID: \"57a0643d-28d1-4d11-bb40-1de33627ca6d\") " pod="openshift-dns/dns-default-b8vzf" Sep 08 04:01:00.702960 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:00.702895 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 04:01:00.702960 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:00.702939 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57a0643d-28d1-4d11-bb40-1de33627ca6d-metrics-tls podName:57a0643d-28d1-4d11-bb40-1de33627ca6d nodeName:}" failed. No retries permitted until 2026-09-08 04:01:01.702927402 +0000 UTC m=+35.625388119 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57a0643d-28d1-4d11-bb40-1de33627ca6d-metrics-tls") pod "dns-default-b8vzf" (UID: "57a0643d-28d1-4d11-bb40-1de33627ca6d") : secret "dns-default-metrics-tls" not found Sep 08 04:01:00.858935 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:00.858910 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pv6qv"] Sep 08 04:01:00.887207 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:01:00.887178 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e2eec1f_b2a7_43d5_a9e7_f15eafb23937.slice/crio-10f67c51e197fdf7eccae5f9f2ca7077d26a0633ea9498d69570d2753054ec56 WatchSource:0}: Error finding container 10f67c51e197fdf7eccae5f9f2ca7077d26a0633ea9498d69570d2753054ec56: Status 404 returned error can't find the container with id 10f67c51e197fdf7eccae5f9f2ca7077d26a0633ea9498d69570d2753054ec56 Sep 08 04:01:01.561806 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:01.561777 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:01:01.562499 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:01.561880 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:01:01.562499 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:01.561962 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:01:01.565739 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:01.565703 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 04:01:01.566472 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:01.566206 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 08 04:01:01.566472 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:01.566270 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-f9485\"" Sep 08 04:01:01.566472 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:01.566208 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-blqwr\"" Sep 08 04:01:01.566472 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:01.566346 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 08 04:01:01.567538 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:01.566896 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 04:01:01.702855 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:01.702832 2574 generic.go:356] "Generic (PLEG): container finished" podID="8ac44940-1e7f-48c1-a9db-8ac81a574f45" containerID="f8fa5b7c20b94a40e9e834a5e772a52db6b0e9d39e5631164b8a7d0474fb0e21" exitCode=0 Sep 08 04:01:01.702962 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:01.702915 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrhb8" event={"ID":"8ac44940-1e7f-48c1-a9db-8ac81a574f45","Type":"ContainerDied","Data":"f8fa5b7c20b94a40e9e834a5e772a52db6b0e9d39e5631164b8a7d0474fb0e21"} Sep 08 04:01:01.704118 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:01.704100 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pv6qv" event={"ID":"3e2eec1f-b2a7-43d5-a9e7-f15eafb23937","Type":"ContainerStarted","Data":"10f67c51e197fdf7eccae5f9f2ca7077d26a0633ea9498d69570d2753054ec56"} Sep 08 04:01:01.711134 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:01.711118 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57a0643d-28d1-4d11-bb40-1de33627ca6d-metrics-tls\") pod \"dns-default-b8vzf\" (UID: \"57a0643d-28d1-4d11-bb40-1de33627ca6d\") " pod="openshift-dns/dns-default-b8vzf" Sep 08 04:01:01.711265 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:01.711250 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 04:01:01.711315 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:01.711301 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57a0643d-28d1-4d11-bb40-1de33627ca6d-metrics-tls podName:57a0643d-28d1-4d11-bb40-1de33627ca6d nodeName:}" failed. No retries permitted until 2026-09-08 04:01:03.711286378 +0000 UTC m=+37.633747096 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57a0643d-28d1-4d11-bb40-1de33627ca6d-metrics-tls") pod "dns-default-b8vzf" (UID: "57a0643d-28d1-4d11-bb40-1de33627ca6d") : secret "dns-default-metrics-tls" not found Sep 08 04:01:02.460662 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:02.460617 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:02.525188 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:02.525149 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:02.709177 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:02.709137 2574 generic.go:356] "Generic (PLEG): container finished" podID="8ac44940-1e7f-48c1-a9db-8ac81a574f45" containerID="ceb4f87d6b453eb2ca742ccb8366771062d50bce508b5cc73d06196552a2bf89" exitCode=0 Sep 08 04:01:02.709498 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:02.709196 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrhb8" event={"ID":"8ac44940-1e7f-48c1-a9db-8ac81a574f45","Type":"ContainerDied","Data":"ceb4f87d6b453eb2ca742ccb8366771062d50bce508b5cc73d06196552a2bf89"} Sep 08 04:01:02.923799 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:02.923772 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:03.051345 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:03.051315 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:03.165649 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:03.165614 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:03.713958 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:03.713929 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrhb8" event={"ID":"8ac44940-1e7f-48c1-a9db-8ac81a574f45","Type":"ContainerStarted","Data":"8467ac68463c8888e0ad6bcb947143f5253efff2de79c1c4c6a00ea8a2e10e66"} Sep 08 04:01:03.724564 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:03.724542 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57a0643d-28d1-4d11-bb40-1de33627ca6d-metrics-tls\") pod \"dns-default-b8vzf\" (UID: \"57a0643d-28d1-4d11-bb40-1de33627ca6d\") " pod="openshift-dns/dns-default-b8vzf" Sep 08 04:01:03.724738 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:03.724716 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 04:01:03.724804 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:03.724790 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57a0643d-28d1-4d11-bb40-1de33627ca6d-metrics-tls podName:57a0643d-28d1-4d11-bb40-1de33627ca6d nodeName:}" failed. No retries permitted until 2026-09-08 04:01:07.724770838 +0000 UTC m=+41.647231570 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57a0643d-28d1-4d11-bb40-1de33627ca6d-metrics-tls") pod "dns-default-b8vzf" (UID: "57a0643d-28d1-4d11-bb40-1de33627ca6d") : secret "dns-default-metrics-tls" not found Sep 08 04:01:03.737295 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:03.737259 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nrhb8" podStartSLOduration=7.90632451 podStartE2EDuration="36.737245449s" totalImagesPullingTime="28.830920939s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 04:00:27 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:03.735760283 +0000 UTC m=+37.658221035" watchObservedRunningTime="2026-09-08 04:01:03.737245449 +0000 UTC m=+37.659706187" Sep 08 04:01:03.830567 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:03.830548 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:03.908570 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:03.908543 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:03.958335 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:03.958302 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pqx2g"] Sep 08 04:01:03.964766 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:03.964704 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-25kxb"] Sep 08 04:01:03.964871 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:03.964807 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pqx2g" Sep 08 04:01:03.967854 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:03.967828 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-k54h4\"" Sep 08 04:01:03.967945 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:03.967911 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 08 04:01:03.967945 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:03.967930 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 08 04:01:03.968597 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:03.968563 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 08 04:01:03.968706 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:03.968602 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 08 04:01:03.972278 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:03.972167 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pqx2g"] Sep 08 04:01:03.972278 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:03.972272 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-25kxb" Sep 08 04:01:03.972885 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:03.972860 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-25kxb"] Sep 08 04:01:03.974843 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:03.974821 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 08 04:01:03.974924 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:03.974854 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 08 04:01:03.975193 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:03.975173 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-p4q9g\"" Sep 08 04:01:03.975291 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:03.975201 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 08 04:01:03.995156 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:03.995134 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:04.027313 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.027289 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b35d6e27-f1ad-4481-a512-36851544fffa-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-pqx2g\" (UID: \"b35d6e27-f1ad-4481-a512-36851544fffa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pqx2g" Sep 08 04:01:04.027417 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.027315 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b35d6e27-f1ad-4481-a512-36851544fffa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-pqx2g\" (UID: \"b35d6e27-f1ad-4481-a512-36851544fffa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pqx2g" Sep 08 04:01:04.027417 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.027334 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfskk\" (UniqueName: \"kubernetes.io/projected/b35d6e27-f1ad-4481-a512-36851544fffa-kube-api-access-lfskk\") pod \"cluster-monitoring-operator-6cdcc96965-pqx2g\" (UID: \"b35d6e27-f1ad-4481-a512-36851544fffa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pqx2g" Sep 08 04:01:04.027417 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.027370 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a8d25d2-4b61-4484-bcba-d9387ebb641f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-25kxb\" (UID: \"3a8d25d2-4b61-4484-bcba-d9387ebb641f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-25kxb" Sep 08 04:01:04.027572 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.027463 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-477r9\" (UniqueName: \"kubernetes.io/projected/3a8d25d2-4b61-4484-bcba-d9387ebb641f-kube-api-access-477r9\") pod \"cluster-samples-operator-7cdcfcf76c-25kxb\" (UID: \"3a8d25d2-4b61-4484-bcba-d9387ebb641f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-25kxb" Sep 08 04:01:04.090704 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.090677 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xmcbr"] Sep 08 04:01:04.109926 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.109894 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xmcbr"] Sep 08 04:01:04.110053 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.110002 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xmcbr" Sep 08 04:01:04.112964 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.112944 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 08 04:01:04.112964 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.112956 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 08 04:01:04.113111 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.113016 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 08 04:01:04.113272 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.113255 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 08 04:01:04.113272 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.113266 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-ppk7x\"" Sep 08 04:01:04.128087 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.128064 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b35d6e27-f1ad-4481-a512-36851544fffa-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-pqx2g\" (UID: \"b35d6e27-f1ad-4481-a512-36851544fffa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pqx2g" Sep 08 04:01:04.128159 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.128090 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b35d6e27-f1ad-4481-a512-36851544fffa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-pqx2g\" (UID: \"b35d6e27-f1ad-4481-a512-36851544fffa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pqx2g" Sep 08 04:01:04.128159 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.128110 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfskk\" (UniqueName: \"kubernetes.io/projected/b35d6e27-f1ad-4481-a512-36851544fffa-kube-api-access-lfskk\") pod \"cluster-monitoring-operator-6cdcc96965-pqx2g\" (UID: \"b35d6e27-f1ad-4481-a512-36851544fffa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pqx2g" Sep 08 04:01:04.128159 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.128127 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh9g9\" (UniqueName: \"kubernetes.io/projected/56994db0-6aa0-44a7-a540-777271c022ee-kube-api-access-vh9g9\") pod \"kube-storage-version-migrator-operator-8d7dd4675-xmcbr\" (UID: \"56994db0-6aa0-44a7-a540-777271c022ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xmcbr" Sep 08 04:01:04.128159 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.128146 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56994db0-6aa0-44a7-a540-777271c022ee-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-xmcbr\" (UID: \"56994db0-6aa0-44a7-a540-777271c022ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xmcbr" Sep 08 04:01:04.128308 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:04.128226 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 04:01:04.128308 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.128240 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a8d25d2-4b61-4484-bcba-d9387ebb641f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-25kxb\" (UID: \"3a8d25d2-4b61-4484-bcba-d9387ebb641f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-25kxb" Sep 08 04:01:04.128308 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.128272 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56994db0-6aa0-44a7-a540-777271c022ee-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-xmcbr\" (UID: \"56994db0-6aa0-44a7-a540-777271c022ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xmcbr" Sep 08 04:01:04.128308 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:04.128285 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b35d6e27-f1ad-4481-a512-36851544fffa-cluster-monitoring-operator-tls podName:b35d6e27-f1ad-4481-a512-36851544fffa nodeName:}" failed. No retries permitted until 2026-09-08 04:01:04.628266138 +0000 UTC m=+38.550726873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b35d6e27-f1ad-4481-a512-36851544fffa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-pqx2g" (UID: "b35d6e27-f1ad-4481-a512-36851544fffa") : secret "cluster-monitoring-operator-tls" not found Sep 08 04:01:04.128519 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:04.128332 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 08 04:01:04.128519 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.128365 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-477r9\" (UniqueName: \"kubernetes.io/projected/3a8d25d2-4b61-4484-bcba-d9387ebb641f-kube-api-access-477r9\") pod \"cluster-samples-operator-7cdcfcf76c-25kxb\" (UID: \"3a8d25d2-4b61-4484-bcba-d9387ebb641f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-25kxb" Sep 08 04:01:04.128519 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:04.128387 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a8d25d2-4b61-4484-bcba-d9387ebb641f-samples-operator-tls podName:3a8d25d2-4b61-4484-bcba-d9387ebb641f nodeName:}" failed. No retries permitted until 2026-09-08 04:01:04.628367616 +0000 UTC m=+38.550828350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3a8d25d2-4b61-4484-bcba-d9387ebb641f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-25kxb" (UID: "3a8d25d2-4b61-4484-bcba-d9387ebb641f") : secret "samples-operator-tls" not found Sep 08 04:01:04.138283 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.138104 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-477r9\" (UniqueName: \"kubernetes.io/projected/3a8d25d2-4b61-4484-bcba-d9387ebb641f-kube-api-access-477r9\") pod \"cluster-samples-operator-7cdcfcf76c-25kxb\" (UID: \"3a8d25d2-4b61-4484-bcba-d9387ebb641f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-25kxb" Sep 08 04:01:04.141536 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.141491 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfskk\" (UniqueName: \"kubernetes.io/projected/b35d6e27-f1ad-4481-a512-36851544fffa-kube-api-access-lfskk\") pod \"cluster-monitoring-operator-6cdcc96965-pqx2g\" (UID: \"b35d6e27-f1ad-4481-a512-36851544fffa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pqx2g" Sep 08 04:01:04.141598 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.141555 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b35d6e27-f1ad-4481-a512-36851544fffa-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-pqx2g\" (UID: \"b35d6e27-f1ad-4481-a512-36851544fffa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pqx2g" Sep 08 04:01:04.171750 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.171729 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-6czj4"] Sep 08 04:01:04.195435 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.195417 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-p59lp"] Sep 08 04:01:04.195600 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.195583 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-6czj4" Sep 08 04:01:04.198030 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.198011 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 08 04:01:04.198114 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.198034 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9wm7x\"" Sep 08 04:01:04.198241 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.198223 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 08 04:01:04.205544 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.205527 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-55766f96d-krtn9"] Sep 08 04:01:04.205639 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.205580 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p59lp" Sep 08 04:01:04.208175 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.208158 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-mt8h7\"" Sep 08 04:01:04.208357 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.208188 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 08 04:01:04.208684 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.208668 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 08 04:01:04.208772 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.208699 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 08 04:01:04.208772 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.208675 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 08 04:01:04.217702 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.217664 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7db884754d-bntvx"] Sep 08 04:01:04.217783 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.217768 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:04.220111 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.220094 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4b7vt\"" Sep 08 04:01:04.220641 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.220626 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 08 04:01:04.220713 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.220646 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 08 04:01:04.220920 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.220905 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 08 04:01:04.220983 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.220919 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 08 04:01:04.221039 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.220983 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 08 04:01:04.221739 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.221718 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 08 04:01:04.229089 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.229067 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e1a7f679-2737-437e-a718-53fe941b999a-default-certificate\") pod \"router-default-55766f96d-krtn9\" (UID: \"e1a7f679-2737-437e-a718-53fe941b999a\") " pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:04.229174 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.229110 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d0fb6dc7-9787-40e0-a9d4-fe3a16f34750-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-6czj4\" (UID: \"d0fb6dc7-9787-40e0-a9d4-fe3a16f34750\") " pod="openshift-network-console/networking-console-plugin-79977f9844-6czj4" Sep 08 04:01:04.229255 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.229199 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc8dj\" (UniqueName: \"kubernetes.io/projected/e1a7f679-2737-437e-a718-53fe941b999a-kube-api-access-wc8dj\") pod \"router-default-55766f96d-krtn9\" (UID: \"e1a7f679-2737-437e-a718-53fe941b999a\") " pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:04.229304 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.229288 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh9g9\" (UniqueName: \"kubernetes.io/projected/56994db0-6aa0-44a7-a540-777271c022ee-kube-api-access-vh9g9\") pod \"kube-storage-version-migrator-operator-8d7dd4675-xmcbr\" (UID: \"56994db0-6aa0-44a7-a540-777271c022ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xmcbr" Sep 08 04:01:04.229353 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.229319 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vjtm\" (UniqueName: \"kubernetes.io/projected/679b83df-ec27-4bf7-902a-a53fc284973d-kube-api-access-2vjtm\") pod \"service-ca-operator-859f94c4b-p59lp\" (UID: \"679b83df-ec27-4bf7-902a-a53fc284973d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p59lp" Sep 08 04:01:04.229353 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.229342 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e1a7f679-2737-437e-a718-53fe941b999a-stats-auth\") pod \"router-default-55766f96d-krtn9\" (UID: \"e1a7f679-2737-437e-a718-53fe941b999a\") " pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:04.229439 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.229371 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56994db0-6aa0-44a7-a540-777271c022ee-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-xmcbr\" (UID: \"56994db0-6aa0-44a7-a540-777271c022ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xmcbr" Sep 08 04:01:04.229485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.229442 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56994db0-6aa0-44a7-a540-777271c022ee-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-xmcbr\" (UID: \"56994db0-6aa0-44a7-a540-777271c022ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xmcbr" Sep 08 04:01:04.229485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.229476 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/679b83df-ec27-4bf7-902a-a53fc284973d-serving-cert\") pod \"service-ca-operator-859f94c4b-p59lp\" (UID: \"679b83df-ec27-4bf7-902a-a53fc284973d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p59lp" Sep 08 04:01:04.229597 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.229523 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/679b83df-ec27-4bf7-902a-a53fc284973d-config\") pod \"service-ca-operator-859f94c4b-p59lp\" (UID: \"679b83df-ec27-4bf7-902a-a53fc284973d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p59lp" Sep 08 04:01:04.229597 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.229545 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d0fb6dc7-9787-40e0-a9d4-fe3a16f34750-nginx-conf\") pod \"networking-console-plugin-79977f9844-6czj4\" (UID: \"d0fb6dc7-9787-40e0-a9d4-fe3a16f34750\") " pod="openshift-network-console/networking-console-plugin-79977f9844-6czj4" Sep 08 04:01:04.229597 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.229567 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1a7f679-2737-437e-a718-53fe941b999a-service-ca-bundle\") pod \"router-default-55766f96d-krtn9\" (UID: \"e1a7f679-2737-437e-a718-53fe941b999a\") " pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:04.229597 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.229588 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1a7f679-2737-437e-a718-53fe941b999a-metrics-certs\") pod \"router-default-55766f96d-krtn9\" (UID: \"e1a7f679-2737-437e-a718-53fe941b999a\") " pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:04.230102 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.230076 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56994db0-6aa0-44a7-a540-777271c022ee-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-xmcbr\" (UID: \"56994db0-6aa0-44a7-a540-777271c022ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xmcbr" Sep 08 04:01:04.231599 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.231576 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56994db0-6aa0-44a7-a540-777271c022ee-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-xmcbr\" (UID: \"56994db0-6aa0-44a7-a540-777271c022ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xmcbr" Sep 08 04:01:04.231720 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.231705 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-6czj4"] Sep 08 04:01:04.231776 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.231728 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-p59lp"] Sep 08 04:01:04.231776 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.231740 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-55766f96d-krtn9"] Sep 08 04:01:04.231776 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.231755 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7db884754d-bntvx"] Sep 08 04:01:04.231913 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.231795 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:04.237226 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.237205 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 08 04:01:04.237226 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.237212 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 08 04:01:04.237363 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.237212 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qx7pf\"" Sep 08 04:01:04.237363 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.237300 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 08 04:01:04.240196 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.240084 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh9g9\" (UniqueName: \"kubernetes.io/projected/56994db0-6aa0-44a7-a540-777271c022ee-kube-api-access-vh9g9\") pod \"kube-storage-version-migrator-operator-8d7dd4675-xmcbr\" (UID: \"56994db0-6aa0-44a7-a540-777271c022ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xmcbr" Sep 08 04:01:04.242127 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.242112 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 08 04:01:04.330160 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.330137 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d0fb6dc7-9787-40e0-a9d4-fe3a16f34750-nginx-conf\") pod \"networking-console-plugin-79977f9844-6czj4\" (UID: \"d0fb6dc7-9787-40e0-a9d4-fe3a16f34750\") " pod="openshift-network-console/networking-console-plugin-79977f9844-6czj4" Sep 08 04:01:04.330244 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.330163 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/679b83df-ec27-4bf7-902a-a53fc284973d-config\") pod \"service-ca-operator-859f94c4b-p59lp\" (UID: \"679b83df-ec27-4bf7-902a-a53fc284973d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p59lp" Sep 08 04:01:04.330244 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.330179 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1a7f679-2737-437e-a718-53fe941b999a-service-ca-bundle\") pod \"router-default-55766f96d-krtn9\" (UID: \"e1a7f679-2737-437e-a718-53fe941b999a\") " pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:04.330244 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.330205 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e1a7f679-2737-437e-a718-53fe941b999a-default-certificate\") pod \"router-default-55766f96d-krtn9\" (UID: \"e1a7f679-2737-437e-a718-53fe941b999a\") " pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:04.330244 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.330229 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-trusted-ca\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:04.330448 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.330244 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-ca-trust-extracted\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:04.330448 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.330270 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc8dj\" (UniqueName: \"kubernetes.io/projected/e1a7f679-2737-437e-a718-53fe941b999a-kube-api-access-wc8dj\") pod \"router-default-55766f96d-krtn9\" (UID: \"e1a7f679-2737-437e-a718-53fe941b999a\") " pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:04.330448 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.330298 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vjtm\" (UniqueName: \"kubernetes.io/projected/679b83df-ec27-4bf7-902a-a53fc284973d-kube-api-access-2vjtm\") pod \"service-ca-operator-859f94c4b-p59lp\" (UID: \"679b83df-ec27-4bf7-902a-a53fc284973d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p59lp" Sep 08 04:01:04.330448 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:04.330324 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e1a7f679-2737-437e-a718-53fe941b999a-service-ca-bundle podName:e1a7f679-2737-437e-a718-53fe941b999a nodeName:}" failed. No retries permitted until 2026-09-08 04:01:04.830304404 +0000 UTC m=+38.752765141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e1a7f679-2737-437e-a718-53fe941b999a-service-ca-bundle") pod "router-default-55766f96d-krtn9" (UID: "e1a7f679-2737-437e-a718-53fe941b999a") : configmap references non-existent config key: service-ca.crt Sep 08 04:01:04.330448 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.330370 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58859\" (UniqueName: \"kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-kube-api-access-58859\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:04.330448 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.330418 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/679b83df-ec27-4bf7-902a-a53fc284973d-serving-cert\") pod \"service-ca-operator-859f94c4b-p59lp\" (UID: \"679b83df-ec27-4bf7-902a-a53fc284973d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p59lp" Sep 08 04:01:04.330448 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.330451 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1a7f679-2737-437e-a718-53fe941b999a-metrics-certs\") pod \"router-default-55766f96d-krtn9\" (UID: \"e1a7f679-2737-437e-a718-53fe941b999a\") " pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:04.330809 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.330474 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-certificates\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:04.330809 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.330498 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-tls\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:04.330809 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.330548 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d0fb6dc7-9787-40e0-a9d4-fe3a16f34750-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-6czj4\" (UID: \"d0fb6dc7-9787-40e0-a9d4-fe3a16f34750\") " pod="openshift-network-console/networking-console-plugin-79977f9844-6czj4" Sep 08 04:01:04.330809 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:04.330573 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 08 04:01:04.330809 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:04.330606 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 08 04:01:04.330809 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:04.330645 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0fb6dc7-9787-40e0-a9d4-fe3a16f34750-networking-console-plugin-cert podName:d0fb6dc7-9787-40e0-a9d4-fe3a16f34750 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:04.830631989 +0000 UTC m=+38.753092723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d0fb6dc7-9787-40e0-a9d4-fe3a16f34750-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-6czj4" (UID: "d0fb6dc7-9787-40e0-a9d4-fe3a16f34750") : secret "networking-console-plugin-cert" not found Sep 08 04:01:04.330809 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.330693 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-installation-pull-secrets\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:04.330809 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:04.330705 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a7f679-2737-437e-a718-53fe941b999a-metrics-certs podName:e1a7f679-2737-437e-a718-53fe941b999a nodeName:}" failed. No retries permitted until 2026-09-08 04:01:04.830692412 +0000 UTC m=+38.753153142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1a7f679-2737-437e-a718-53fe941b999a-metrics-certs") pod "router-default-55766f96d-krtn9" (UID: "e1a7f679-2737-437e-a718-53fe941b999a") : secret "router-metrics-certs-default" not found Sep 08 04:01:04.330809 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.330733 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-bound-sa-token\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:04.330809 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.330768 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e1a7f679-2737-437e-a718-53fe941b999a-stats-auth\") pod \"router-default-55766f96d-krtn9\" (UID: \"e1a7f679-2737-437e-a718-53fe941b999a\") " pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:04.331198 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.330816 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-image-registry-private-configuration\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:04.331198 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.331000 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d0fb6dc7-9787-40e0-a9d4-fe3a16f34750-nginx-conf\") pod \"networking-console-plugin-79977f9844-6czj4\" (UID: \"d0fb6dc7-9787-40e0-a9d4-fe3a16f34750\") " pod="openshift-network-console/networking-console-plugin-79977f9844-6czj4" Sep 08 04:01:04.331198 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.331044 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/679b83df-ec27-4bf7-902a-a53fc284973d-config\") pod \"service-ca-operator-859f94c4b-p59lp\" (UID: \"679b83df-ec27-4bf7-902a-a53fc284973d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p59lp" Sep 08 04:01:04.332548 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.332529 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e1a7f679-2737-437e-a718-53fe941b999a-default-certificate\") pod \"router-default-55766f96d-krtn9\" (UID: \"e1a7f679-2737-437e-a718-53fe941b999a\") " pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:04.332626 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.332567 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/679b83df-ec27-4bf7-902a-a53fc284973d-serving-cert\") pod \"service-ca-operator-859f94c4b-p59lp\" (UID: \"679b83df-ec27-4bf7-902a-a53fc284973d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p59lp" Sep 08 04:01:04.332767 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.332751 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e1a7f679-2737-437e-a718-53fe941b999a-stats-auth\") pod \"router-default-55766f96d-krtn9\" (UID: \"e1a7f679-2737-437e-a718-53fe941b999a\") " pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:04.338812 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.338790 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vjtm\" (UniqueName: \"kubernetes.io/projected/679b83df-ec27-4bf7-902a-a53fc284973d-kube-api-access-2vjtm\") pod \"service-ca-operator-859f94c4b-p59lp\" (UID: \"679b83df-ec27-4bf7-902a-a53fc284973d\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p59lp" Sep 08 04:01:04.339064 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.339048 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc8dj\" (UniqueName: \"kubernetes.io/projected/e1a7f679-2737-437e-a718-53fe941b999a-kube-api-access-wc8dj\") pod \"router-default-55766f96d-krtn9\" (UID: \"e1a7f679-2737-437e-a718-53fe941b999a\") " pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:04.353023 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.352984 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:04.418867 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.418848 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xmcbr" Sep 08 04:01:04.431249 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.431230 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-image-registry-private-configuration\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:04.431314 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.431302 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-trusted-ca\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:04.431347 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.431320 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-ca-trust-extracted\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:04.431347 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.431341 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58859\" (UniqueName: \"kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-kube-api-access-58859\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:04.431444 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.431420 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-certificates\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:04.431474 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.431464 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-tls\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:04.431587 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.431565 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-installation-pull-secrets\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:04.431587 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.431591 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-bound-sa-token\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:04.431754 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:04.431675 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 04:01:04.431754 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:04.431691 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db884754d-bntvx: secret "image-registry-tls" not found Sep 08 04:01:04.431863 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:04.431760 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-tls podName:b0ce25c1-3ba6-4f70-9457-7ded8bc28360 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:04.931745694 +0000 UTC m=+38.854206412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-tls") pod "image-registry-7db884754d-bntvx" (UID: "b0ce25c1-3ba6-4f70-9457-7ded8bc28360") : secret "image-registry-tls" not found Sep 08 04:01:04.431863 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.431840 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-ca-trust-extracted\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:04.432136 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.432101 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-certificates\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:04.432290 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.432271 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-trusted-ca\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:04.433268 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.433250 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-image-registry-private-configuration\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:04.433514 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.433488 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-installation-pull-secrets\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:04.441610 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.441582 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58859\" (UniqueName: \"kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-kube-api-access-58859\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:04.441992 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.441977 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-bound-sa-token\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:04.514533 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.514494 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p59lp" Sep 08 04:01:04.633109 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.633081 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b35d6e27-f1ad-4481-a512-36851544fffa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-pqx2g\" (UID: \"b35d6e27-f1ad-4481-a512-36851544fffa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pqx2g" Sep 08 04:01:04.633218 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.633115 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a8d25d2-4b61-4484-bcba-d9387ebb641f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-25kxb\" (UID: \"3a8d25d2-4b61-4484-bcba-d9387ebb641f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-25kxb" Sep 08 04:01:04.633267 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:04.633231 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 04:01:04.633298 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:04.633283 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 08 04:01:04.633298 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:04.633295 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b35d6e27-f1ad-4481-a512-36851544fffa-cluster-monitoring-operator-tls podName:b35d6e27-f1ad-4481-a512-36851544fffa nodeName:}" failed. No retries permitted until 2026-09-08 04:01:05.633279644 +0000 UTC m=+39.555740377 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b35d6e27-f1ad-4481-a512-36851544fffa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-pqx2g" (UID: "b35d6e27-f1ad-4481-a512-36851544fffa") : secret "cluster-monitoring-operator-tls" not found Sep 08 04:01:04.633414 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:04.633322 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a8d25d2-4b61-4484-bcba-d9387ebb641f-samples-operator-tls podName:3a8d25d2-4b61-4484-bcba-d9387ebb641f nodeName:}" failed. No retries permitted until 2026-09-08 04:01:05.633311104 +0000 UTC m=+39.555771842 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3a8d25d2-4b61-4484-bcba-d9387ebb641f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-25kxb" (UID: "3a8d25d2-4b61-4484-bcba-d9387ebb641f") : secret "samples-operator-tls" not found Sep 08 04:01:04.710400 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.710373 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xmcbr"] Sep 08 04:01:04.713167 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:01:04.713136 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56994db0_6aa0_44a7_a540_777271c022ee.slice/crio-0493a9ca98cafbd8176adf7fd81b03241045ef41d534ad9597f67da5e6594708 WatchSource:0}: Error finding container 0493a9ca98cafbd8176adf7fd81b03241045ef41d534ad9597f67da5e6594708: Status 404 returned error can't find the container with id 0493a9ca98cafbd8176adf7fd81b03241045ef41d534ad9597f67da5e6594708 Sep 08 04:01:04.719163 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.719135 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pv6qv" event={"ID":"3e2eec1f-b2a7-43d5-a9e7-f15eafb23937","Type":"ContainerStarted","Data":"7583e2c205498a0369dd55b11013f6dcf6df63a7b6e80926a0216bbd9d4dc368"} Sep 08 04:01:04.720667 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.720642 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xmcbr" event={"ID":"56994db0-6aa0-44a7-a540-777271c022ee","Type":"ContainerStarted","Data":"0493a9ca98cafbd8176adf7fd81b03241045ef41d534ad9597f67da5e6594708"} Sep 08 04:01:04.735114 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.735075 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pv6qv" podStartSLOduration=2.779787611 podStartE2EDuration="5.735060967s" totalImagesPullingTime="2.955273356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:00:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:04.734187399 +0000 UTC m=+38.656648158" watchObservedRunningTime="2026-09-08 04:01:04.735060967 +0000 UTC m=+38.657521705" Sep 08 04:01:04.824077 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.824055 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-p59lp"] Sep 08 04:01:04.826475 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:01:04.826450 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679b83df_ec27_4bf7_902a_a53fc284973d.slice/crio-e0ba0fa15b39591465d3c83aefc2f121ca7c945c2ecc0d0d379b28180cb44159 WatchSource:0}: Error finding container e0ba0fa15b39591465d3c83aefc2f121ca7c945c2ecc0d0d379b28180cb44159: Status 404 returned error can't find the container with id e0ba0fa15b39591465d3c83aefc2f121ca7c945c2ecc0d0d379b28180cb44159 Sep 08 04:01:04.834744 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.834725 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1a7f679-2737-437e-a718-53fe941b999a-metrics-certs\") pod \"router-default-55766f96d-krtn9\" (UID: \"e1a7f679-2737-437e-a718-53fe941b999a\") " pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:04.834826 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.834763 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d0fb6dc7-9787-40e0-a9d4-fe3a16f34750-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-6czj4\" (UID: \"d0fb6dc7-9787-40e0-a9d4-fe3a16f34750\") " pod="openshift-network-console/networking-console-plugin-79977f9844-6czj4" Sep 08 04:01:04.834871 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:04.834853 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 08 04:01:04.834919 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:04.834906 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a7f679-2737-437e-a718-53fe941b999a-metrics-certs podName:e1a7f679-2737-437e-a718-53fe941b999a nodeName:}" failed. No retries permitted until 2026-09-08 04:01:05.834888666 +0000 UTC m=+39.757349384 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1a7f679-2737-437e-a718-53fe941b999a-metrics-certs") pod "router-default-55766f96d-krtn9" (UID: "e1a7f679-2737-437e-a718-53fe941b999a") : secret "router-metrics-certs-default" not found Sep 08 04:01:04.834967 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:04.834921 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 08 04:01:04.835008 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:04.834967 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0fb6dc7-9787-40e0-a9d4-fe3a16f34750-networking-console-plugin-cert podName:d0fb6dc7-9787-40e0-a9d4-fe3a16f34750 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:05.834952702 +0000 UTC m=+39.757413441 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d0fb6dc7-9787-40e0-a9d4-fe3a16f34750-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-6czj4" (UID: "d0fb6dc7-9787-40e0-a9d4-fe3a16f34750") : secret "networking-console-plugin-cert" not found Sep 08 04:01:04.835008 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.834961 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1a7f679-2737-437e-a718-53fe941b999a-service-ca-bundle\") pod \"router-default-55766f96d-krtn9\" (UID: \"e1a7f679-2737-437e-a718-53fe941b999a\") " pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:04.835073 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:04.835035 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e1a7f679-2737-437e-a718-53fe941b999a-service-ca-bundle podName:e1a7f679-2737-437e-a718-53fe941b999a nodeName:}" failed. No retries permitted until 2026-09-08 04:01:05.835024325 +0000 UTC m=+39.757485056 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e1a7f679-2737-437e-a718-53fe941b999a-service-ca-bundle") pod "router-default-55766f96d-krtn9" (UID: "e1a7f679-2737-437e-a718-53fe941b999a") : configmap references non-existent config key: service-ca.crt Sep 08 04:01:04.935893 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:04.935870 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-tls\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:04.936012 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:04.935996 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 04:01:04.936065 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:04.936014 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db884754d-bntvx: secret "image-registry-tls" not found Sep 08 04:01:04.936065 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:04.936052 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-tls podName:b0ce25c1-3ba6-4f70-9457-7ded8bc28360 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:05.936041729 +0000 UTC m=+39.858502446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-tls") pod "image-registry-7db884754d-bntvx" (UID: "b0ce25c1-3ba6-4f70-9457-7ded8bc28360") : secret "image-registry-tls" not found Sep 08 04:01:05.520093 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:05.520064 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:05.586129 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:05.586104 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:05.642320 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:05.642298 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b35d6e27-f1ad-4481-a512-36851544fffa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-pqx2g\" (UID: \"b35d6e27-f1ad-4481-a512-36851544fffa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pqx2g" Sep 08 04:01:05.642420 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:05.642329 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a8d25d2-4b61-4484-bcba-d9387ebb641f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-25kxb\" (UID: \"3a8d25d2-4b61-4484-bcba-d9387ebb641f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-25kxb" Sep 08 04:01:05.642476 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:05.642463 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 04:01:05.642537 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:05.642486 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 08 04:01:05.642596 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:05.642542 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b35d6e27-f1ad-4481-a512-36851544fffa-cluster-monitoring-operator-tls podName:b35d6e27-f1ad-4481-a512-36851544fffa nodeName:}" failed. No retries permitted until 2026-09-08 04:01:07.642525233 +0000 UTC m=+41.564985964 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b35d6e27-f1ad-4481-a512-36851544fffa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-pqx2g" (UID: "b35d6e27-f1ad-4481-a512-36851544fffa") : secret "cluster-monitoring-operator-tls" not found Sep 08 04:01:05.642596 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:05.642571 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a8d25d2-4b61-4484-bcba-d9387ebb641f-samples-operator-tls podName:3a8d25d2-4b61-4484-bcba-d9387ebb641f nodeName:}" failed. No retries permitted until 2026-09-08 04:01:07.642552133 +0000 UTC m=+41.565012850 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3a8d25d2-4b61-4484-bcba-d9387ebb641f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-25kxb" (UID: "3a8d25d2-4b61-4484-bcba-d9387ebb641f") : secret "samples-operator-tls" not found Sep 08 04:01:05.724704 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:05.724675 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p59lp" event={"ID":"679b83df-ec27-4bf7-902a-a53fc284973d","Type":"ContainerStarted","Data":"e0ba0fa15b39591465d3c83aefc2f121ca7c945c2ecc0d0d379b28180cb44159"} Sep 08 04:01:05.821118 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:05.821047 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:05.844050 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:05.844022 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1a7f679-2737-437e-a718-53fe941b999a-metrics-certs\") pod \"router-default-55766f96d-krtn9\" (UID: \"e1a7f679-2737-437e-a718-53fe941b999a\") " pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:05.844174 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:05.844086 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d0fb6dc7-9787-40e0-a9d4-fe3a16f34750-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-6czj4\" (UID: \"d0fb6dc7-9787-40e0-a9d4-fe3a16f34750\") " pod="openshift-network-console/networking-console-plugin-79977f9844-6czj4" Sep 08 04:01:05.844228 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:05.844176 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 08 04:01:05.844228 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:05.844184 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1a7f679-2737-437e-a718-53fe941b999a-service-ca-bundle\") pod \"router-default-55766f96d-krtn9\" (UID: \"e1a7f679-2737-437e-a718-53fe941b999a\") " pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:05.844313 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:05.844253 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a7f679-2737-437e-a718-53fe941b999a-metrics-certs podName:e1a7f679-2737-437e-a718-53fe941b999a nodeName:}" failed. No retries permitted until 2026-09-08 04:01:07.844235223 +0000 UTC m=+41.766695953 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1a7f679-2737-437e-a718-53fe941b999a-metrics-certs") pod "router-default-55766f96d-krtn9" (UID: "e1a7f679-2737-437e-a718-53fe941b999a") : secret "router-metrics-certs-default" not found Sep 08 04:01:05.844365 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:05.844335 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e1a7f679-2737-437e-a718-53fe941b999a-service-ca-bundle podName:e1a7f679-2737-437e-a718-53fe941b999a nodeName:}" failed. No retries permitted until 2026-09-08 04:01:07.844319777 +0000 UTC m=+41.766780500 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e1a7f679-2737-437e-a718-53fe941b999a-service-ca-bundle") pod "router-default-55766f96d-krtn9" (UID: "e1a7f679-2737-437e-a718-53fe941b999a") : configmap references non-existent config key: service-ca.crt Sep 08 04:01:05.844419 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:05.844406 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 08 04:01:05.844467 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:05.844447 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0fb6dc7-9787-40e0-a9d4-fe3a16f34750-networking-console-plugin-cert podName:d0fb6dc7-9787-40e0-a9d4-fe3a16f34750 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:07.84443594 +0000 UTC m=+41.766896671 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d0fb6dc7-9787-40e0-a9d4-fe3a16f34750-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-6czj4" (UID: "d0fb6dc7-9787-40e0-a9d4-fe3a16f34750") : secret "networking-console-plugin-cert" not found Sep 08 04:01:05.923105 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:05.923076 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:05.945143 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:05.945110 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-tls\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:05.945267 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:05.945249 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 04:01:05.945336 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:05.945271 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db884754d-bntvx: secret "image-registry-tls" not found Sep 08 04:01:05.945395 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:05.945341 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-tls podName:b0ce25c1-3ba6-4f70-9457-7ded8bc28360 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:07.94532386 +0000 UTC m=+41.867784578 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-tls") pod "image-registry-7db884754d-bntvx" (UID: "b0ce25c1-3ba6-4f70-9457-7ded8bc28360") : secret "image-registry-tls" not found Sep 08 04:01:06.207620 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:06.207541 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:06.280167 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:06.280134 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:06.880772 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:06.880737 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:07.660071 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:07.660044 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b35d6e27-f1ad-4481-a512-36851544fffa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-pqx2g\" (UID: \"b35d6e27-f1ad-4481-a512-36851544fffa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pqx2g" Sep 08 04:01:07.660227 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:07.660078 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a8d25d2-4b61-4484-bcba-d9387ebb641f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-25kxb\" (UID: \"3a8d25d2-4b61-4484-bcba-d9387ebb641f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-25kxb" Sep 08 04:01:07.660227 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:07.660197 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 08 04:01:07.660303 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:07.660198 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 04:01:07.660303 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:07.660266 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a8d25d2-4b61-4484-bcba-d9387ebb641f-samples-operator-tls podName:3a8d25d2-4b61-4484-bcba-d9387ebb641f nodeName:}" failed. No retries permitted until 2026-09-08 04:01:11.660246289 +0000 UTC m=+45.582707028 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3a8d25d2-4b61-4484-bcba-d9387ebb641f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-25kxb" (UID: "3a8d25d2-4b61-4484-bcba-d9387ebb641f") : secret "samples-operator-tls" not found Sep 08 04:01:07.660303 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:07.660287 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b35d6e27-f1ad-4481-a512-36851544fffa-cluster-monitoring-operator-tls podName:b35d6e27-f1ad-4481-a512-36851544fffa nodeName:}" failed. No retries permitted until 2026-09-08 04:01:11.660277189 +0000 UTC m=+45.582737906 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b35d6e27-f1ad-4481-a512-36851544fffa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-pqx2g" (UID: "b35d6e27-f1ad-4481-a512-36851544fffa") : secret "cluster-monitoring-operator-tls" not found Sep 08 04:01:07.760802 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:07.760771 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57a0643d-28d1-4d11-bb40-1de33627ca6d-metrics-tls\") pod \"dns-default-b8vzf\" (UID: \"57a0643d-28d1-4d11-bb40-1de33627ca6d\") " pod="openshift-dns/dns-default-b8vzf" Sep 08 04:01:07.760970 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:07.760900 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 04:01:07.761028 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:07.760973 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57a0643d-28d1-4d11-bb40-1de33627ca6d-metrics-tls podName:57a0643d-28d1-4d11-bb40-1de33627ca6d nodeName:}" failed. No retries permitted until 2026-09-08 04:01:15.760952966 +0000 UTC m=+49.683413726 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57a0643d-28d1-4d11-bb40-1de33627ca6d-metrics-tls") pod "dns-default-b8vzf" (UID: "57a0643d-28d1-4d11-bb40-1de33627ca6d") : secret "dns-default-metrics-tls" not found Sep 08 04:01:07.861584 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:07.861550 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1a7f679-2737-437e-a718-53fe941b999a-metrics-certs\") pod \"router-default-55766f96d-krtn9\" (UID: \"e1a7f679-2737-437e-a718-53fe941b999a\") " pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:07.861735 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:07.861617 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d0fb6dc7-9787-40e0-a9d4-fe3a16f34750-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-6czj4\" (UID: \"d0fb6dc7-9787-40e0-a9d4-fe3a16f34750\") " pod="openshift-network-console/networking-console-plugin-79977f9844-6czj4" Sep 08 04:01:07.861735 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:07.861699 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1a7f679-2737-437e-a718-53fe941b999a-service-ca-bundle\") pod \"router-default-55766f96d-krtn9\" (UID: \"e1a7f679-2737-437e-a718-53fe941b999a\") " pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:07.861735 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:07.861708 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 08 04:01:07.861878 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:07.861708 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 08 04:01:07.861878 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:07.861772 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0fb6dc7-9787-40e0-a9d4-fe3a16f34750-networking-console-plugin-cert podName:d0fb6dc7-9787-40e0-a9d4-fe3a16f34750 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:11.861754149 +0000 UTC m=+45.784214867 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d0fb6dc7-9787-40e0-a9d4-fe3a16f34750-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-6czj4" (UID: "d0fb6dc7-9787-40e0-a9d4-fe3a16f34750") : secret "networking-console-plugin-cert" not found Sep 08 04:01:07.861878 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:07.861833 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e1a7f679-2737-437e-a718-53fe941b999a-service-ca-bundle podName:e1a7f679-2737-437e-a718-53fe941b999a nodeName:}" failed. No retries permitted until 2026-09-08 04:01:11.861813257 +0000 UTC m=+45.784273990 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e1a7f679-2737-437e-a718-53fe941b999a-service-ca-bundle") pod "router-default-55766f96d-krtn9" (UID: "e1a7f679-2737-437e-a718-53fe941b999a") : configmap references non-existent config key: service-ca.crt Sep 08 04:01:07.861878 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:07.861849 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a7f679-2737-437e-a718-53fe941b999a-metrics-certs podName:e1a7f679-2737-437e-a718-53fe941b999a nodeName:}" failed. No retries permitted until 2026-09-08 04:01:11.861839406 +0000 UTC m=+45.784300130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1a7f679-2737-437e-a718-53fe941b999a-metrics-certs") pod "router-default-55766f96d-krtn9" (UID: "e1a7f679-2737-437e-a718-53fe941b999a") : secret "router-metrics-certs-default" not found Sep 08 04:01:07.962797 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:07.962715 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-tls\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:07.963188 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:07.962877 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 04:01:07.963188 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:07.962894 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db884754d-bntvx: secret "image-registry-tls" not found Sep 08 04:01:07.963188 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:07.962947 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-tls podName:b0ce25c1-3ba6-4f70-9457-7ded8bc28360 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:11.962928607 +0000 UTC m=+45.885389341 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-tls") pod "image-registry-7db884754d-bntvx" (UID: "b0ce25c1-3ba6-4f70-9457-7ded8bc28360") : secret "image-registry-tls" not found Sep 08 04:01:08.254645 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:08.254621 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:08.354191 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:08.354170 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:08.734363 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:08.734295 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xmcbr" event={"ID":"56994db0-6aa0-44a7-a540-777271c022ee","Type":"ContainerStarted","Data":"714d78de5b7ae9ed6803aff29e66dda2b0dd01fcd5d09d027def1bf9894ffea5"} Sep 08 04:01:08.735842 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:08.735817 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p59lp" event={"ID":"679b83df-ec27-4bf7-902a-a53fc284973d","Type":"ContainerStarted","Data":"f290a1c1387b5df3bf04fd0b25375a9dc8f9a377a0a57c1a2c3c70d67a865bee"} Sep 08 04:01:08.768488 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:08.768439 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xmcbr" podStartSLOduration=1.395078747 podStartE2EDuration="4.768420509s" totalImagesPullingTime="3.373341762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:08.75205976 +0000 UTC m=+42.674520517" watchObservedRunningTime="2026-09-08 04:01:08.768420509 +0000 UTC m=+42.690881249" Sep 08 04:01:08.773865 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:08.771054 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p59lp" podStartSLOduration=1.254046823 podStartE2EDuration="4.77101132s" totalImagesPullingTime="3.516964497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:08.768012289 +0000 UTC m=+42.690473050" watchObservedRunningTime="2026-09-08 04:01:08.77101132 +0000 UTC m=+42.693472065" Sep 08 04:01:09.305369 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:09.305319 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:10.323796 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:10.323760 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:10.415984 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:10.415954 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:11.055040 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.055015 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kjg2p"] Sep 08 04:01:11.075057 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.075032 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:11.086115 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.086097 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:11.089549 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.089529 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 08 04:01:11.090193 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.090176 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 08 04:01:11.097013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.096995 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7wgqk\"" Sep 08 04:01:11.097436 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.097421 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 08 04:01:11.097614 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.097601 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 08 04:01:11.109437 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.109416 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kjg2p"] Sep 08 04:01:11.186748 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.186729 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b5cda57-838f-4ba0-87be-f09808d06d65-data-volume\") pod \"insights-runtime-extractor-kjg2p\" (UID: \"5b5cda57-838f-4ba0-87be-f09808d06d65\") " pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:11.186840 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.186780 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sthn\" (UniqueName: \"kubernetes.io/projected/5b5cda57-838f-4ba0-87be-f09808d06d65-kube-api-access-8sthn\") pod \"insights-runtime-extractor-kjg2p\" (UID: \"5b5cda57-838f-4ba0-87be-f09808d06d65\") " pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:11.186894 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.186839 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b5cda57-838f-4ba0-87be-f09808d06d65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kjg2p\" (UID: \"5b5cda57-838f-4ba0-87be-f09808d06d65\") " pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:11.186947 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.186925 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b5cda57-838f-4ba0-87be-f09808d06d65-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kjg2p\" (UID: \"5b5cda57-838f-4ba0-87be-f09808d06d65\") " pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:11.186999 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.186979 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5b5cda57-838f-4ba0-87be-f09808d06d65-crio-socket\") pod \"insights-runtime-extractor-kjg2p\" (UID: \"5b5cda57-838f-4ba0-87be-f09808d06d65\") " pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:11.199111 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.199088 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:11.287344 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.287318 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sthn\" (UniqueName: \"kubernetes.io/projected/5b5cda57-838f-4ba0-87be-f09808d06d65-kube-api-access-8sthn\") pod \"insights-runtime-extractor-kjg2p\" (UID: \"5b5cda57-838f-4ba0-87be-f09808d06d65\") " pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:11.287442 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.287374 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b5cda57-838f-4ba0-87be-f09808d06d65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kjg2p\" (UID: \"5b5cda57-838f-4ba0-87be-f09808d06d65\") " pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:11.287527 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:11.287469 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 04:01:11.287589 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.287571 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b5cda57-838f-4ba0-87be-f09808d06d65-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kjg2p\" (UID: \"5b5cda57-838f-4ba0-87be-f09808d06d65\") " pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:11.287648 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:11.287609 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b5cda57-838f-4ba0-87be-f09808d06d65-insights-runtime-extractor-tls podName:5b5cda57-838f-4ba0-87be-f09808d06d65 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:11.787593767 +0000 UTC m=+45.710054485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5b5cda57-838f-4ba0-87be-f09808d06d65-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kjg2p" (UID: "5b5cda57-838f-4ba0-87be-f09808d06d65") : secret "insights-runtime-extractor-tls" not found Sep 08 04:01:11.287648 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:11.287637 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 04:01:11.287648 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.287645 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5b5cda57-838f-4ba0-87be-f09808d06d65-crio-socket\") pod \"insights-runtime-extractor-kjg2p\" (UID: \"5b5cda57-838f-4ba0-87be-f09808d06d65\") " pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:11.287788 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.287689 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b5cda57-838f-4ba0-87be-f09808d06d65-data-volume\") pod \"insights-runtime-extractor-kjg2p\" (UID: \"5b5cda57-838f-4ba0-87be-f09808d06d65\") " pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:11.287788 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:11.287729 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b5cda57-838f-4ba0-87be-f09808d06d65-kube-rbac-proxy-cm podName:5b5cda57-838f-4ba0-87be-f09808d06d65 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:11.787708 +0000 UTC m=+45.710168732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5b5cda57-838f-4ba0-87be-f09808d06d65-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kjg2p" (UID: "5b5cda57-838f-4ba0-87be-f09808d06d65") : configmap "kube-rbac-proxy" not found Sep 08 04:01:11.287856 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.287794 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5b5cda57-838f-4ba0-87be-f09808d06d65-crio-socket\") pod \"insights-runtime-extractor-kjg2p\" (UID: \"5b5cda57-838f-4ba0-87be-f09808d06d65\") " pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:11.287953 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.287936 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b5cda57-838f-4ba0-87be-f09808d06d65-data-volume\") pod \"insights-runtime-extractor-kjg2p\" (UID: \"5b5cda57-838f-4ba0-87be-f09808d06d65\") " pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:11.298435 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.298415 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sthn\" (UniqueName: \"kubernetes.io/projected/5b5cda57-838f-4ba0-87be-f09808d06d65-kube-api-access-8sthn\") pod \"insights-runtime-extractor-kjg2p\" (UID: \"5b5cda57-838f-4ba0-87be-f09808d06d65\") " pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:11.691249 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.691228 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a8d25d2-4b61-4484-bcba-d9387ebb641f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-25kxb\" (UID: \"3a8d25d2-4b61-4484-bcba-d9387ebb641f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-25kxb" Sep 08 04:01:11.691621 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.691355 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b35d6e27-f1ad-4481-a512-36851544fffa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-pqx2g\" (UID: \"b35d6e27-f1ad-4481-a512-36851544fffa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pqx2g" Sep 08 04:01:11.691621 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:11.691387 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 08 04:01:11.691621 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:11.691430 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 04:01:11.691621 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:11.691480 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a8d25d2-4b61-4484-bcba-d9387ebb641f-samples-operator-tls podName:3a8d25d2-4b61-4484-bcba-d9387ebb641f nodeName:}" failed. No retries permitted until 2026-09-08 04:01:19.691460372 +0000 UTC m=+53.613921109 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3a8d25d2-4b61-4484-bcba-d9387ebb641f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-25kxb" (UID: "3a8d25d2-4b61-4484-bcba-d9387ebb641f") : secret "samples-operator-tls" not found Sep 08 04:01:11.691621 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:11.691521 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b35d6e27-f1ad-4481-a512-36851544fffa-cluster-monitoring-operator-tls podName:b35d6e27-f1ad-4481-a512-36851544fffa nodeName:}" failed. No retries permitted until 2026-09-08 04:01:19.691491212 +0000 UTC m=+53.613951933 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b35d6e27-f1ad-4481-a512-36851544fffa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-pqx2g" (UID: "b35d6e27-f1ad-4481-a512-36851544fffa") : secret "cluster-monitoring-operator-tls" not found Sep 08 04:01:11.754430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.754405 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:11.792099 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.792077 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b5cda57-838f-4ba0-87be-f09808d06d65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kjg2p\" (UID: \"5b5cda57-838f-4ba0-87be-f09808d06d65\") " pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:11.792211 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.792124 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b5cda57-838f-4ba0-87be-f09808d06d65-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kjg2p\" (UID: \"5b5cda57-838f-4ba0-87be-f09808d06d65\") " pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:11.792270 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:11.792225 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 04:01:11.792321 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:11.792282 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b5cda57-838f-4ba0-87be-f09808d06d65-insights-runtime-extractor-tls podName:5b5cda57-838f-4ba0-87be-f09808d06d65 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:12.792262916 +0000 UTC m=+46.714723685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5b5cda57-838f-4ba0-87be-f09808d06d65-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kjg2p" (UID: "5b5cda57-838f-4ba0-87be-f09808d06d65") : secret "insights-runtime-extractor-tls" not found Sep 08 04:01:11.792321 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:11.792311 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 04:01:11.792485 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:11.792353 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b5cda57-838f-4ba0-87be-f09808d06d65-kube-rbac-proxy-cm podName:5b5cda57-838f-4ba0-87be-f09808d06d65 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:12.792342308 +0000 UTC m=+46.714803025 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5b5cda57-838f-4ba0-87be-f09808d06d65-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kjg2p" (UID: "5b5cda57-838f-4ba0-87be-f09808d06d65") : configmap "kube-rbac-proxy" not found Sep 08 04:01:11.846478 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.846459 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:11.893308 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.893283 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1a7f679-2737-437e-a718-53fe941b999a-service-ca-bundle\") pod \"router-default-55766f96d-krtn9\" (UID: \"e1a7f679-2737-437e-a718-53fe941b999a\") " pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:11.893400 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.893351 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1a7f679-2737-437e-a718-53fe941b999a-metrics-certs\") pod \"router-default-55766f96d-krtn9\" (UID: \"e1a7f679-2737-437e-a718-53fe941b999a\") " pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:11.893400 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.893382 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d0fb6dc7-9787-40e0-a9d4-fe3a16f34750-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-6czj4\" (UID: \"d0fb6dc7-9787-40e0-a9d4-fe3a16f34750\") " pod="openshift-network-console/networking-console-plugin-79977f9844-6czj4" Sep 08 04:01:11.893542 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:11.893445 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e1a7f679-2737-437e-a718-53fe941b999a-service-ca-bundle podName:e1a7f679-2737-437e-a718-53fe941b999a nodeName:}" failed. No retries permitted until 2026-09-08 04:01:19.893428179 +0000 UTC m=+53.815888945 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e1a7f679-2737-437e-a718-53fe941b999a-service-ca-bundle") pod "router-default-55766f96d-krtn9" (UID: "e1a7f679-2737-437e-a718-53fe941b999a") : configmap references non-existent config key: service-ca.crt Sep 08 04:01:11.893542 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:11.893474 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 08 04:01:11.893542 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:11.893487 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 08 04:01:11.893542 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:11.893540 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0fb6dc7-9787-40e0-a9d4-fe3a16f34750-networking-console-plugin-cert podName:d0fb6dc7-9787-40e0-a9d4-fe3a16f34750 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:19.893529361 +0000 UTC m=+53.815990078 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d0fb6dc7-9787-40e0-a9d4-fe3a16f34750-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-6czj4" (UID: "d0fb6dc7-9787-40e0-a9d4-fe3a16f34750") : secret "networking-console-plugin-cert" not found Sep 08 04:01:11.893702 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:11.893553 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a7f679-2737-437e-a718-53fe941b999a-metrics-certs podName:e1a7f679-2737-437e-a718-53fe941b999a nodeName:}" failed. No retries permitted until 2026-09-08 04:01:19.893547303 +0000 UTC m=+53.816008020 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1a7f679-2737-437e-a718-53fe941b999a-metrics-certs") pod "router-default-55766f96d-krtn9" (UID: "e1a7f679-2737-437e-a718-53fe941b999a") : secret "router-metrics-certs-default" not found Sep 08 04:01:11.994568 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:11.994499 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-tls\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:11.994646 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:11.994626 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 04:01:11.994646 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:11.994642 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db884754d-bntvx: secret "image-registry-tls" not found Sep 08 04:01:11.994711 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:11.994680 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-tls podName:b0ce25c1-3ba6-4f70-9457-7ded8bc28360 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:19.994670217 +0000 UTC m=+53.917130934 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-tls") pod "image-registry-7db884754d-bntvx" (UID: "b0ce25c1-3ba6-4f70-9457-7ded8bc28360") : secret "image-registry-tls" not found Sep 08 04:01:12.690280 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:12.690254 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:12.801801 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:12.801773 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b5cda57-838f-4ba0-87be-f09808d06d65-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kjg2p\" (UID: \"5b5cda57-838f-4ba0-87be-f09808d06d65\") " pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:12.802132 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:12.801887 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 04:01:12.802132 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:12.801942 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 04:01:12.802132 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:12.801950 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b5cda57-838f-4ba0-87be-f09808d06d65-kube-rbac-proxy-cm podName:5b5cda57-838f-4ba0-87be-f09808d06d65 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:14.801936412 +0000 UTC m=+48.724397129 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5b5cda57-838f-4ba0-87be-f09808d06d65-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kjg2p" (UID: "5b5cda57-838f-4ba0-87be-f09808d06d65") : configmap "kube-rbac-proxy" not found Sep 08 04:01:12.802132 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:12.801886 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b5cda57-838f-4ba0-87be-f09808d06d65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kjg2p\" (UID: \"5b5cda57-838f-4ba0-87be-f09808d06d65\") " pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:12.802132 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:12.801978 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b5cda57-838f-4ba0-87be-f09808d06d65-insights-runtime-extractor-tls podName:5b5cda57-838f-4ba0-87be-f09808d06d65 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:14.801967534 +0000 UTC m=+48.724428251 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5b5cda57-838f-4ba0-87be-f09808d06d65-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kjg2p" (UID: "5b5cda57-838f-4ba0-87be-f09808d06d65") : secret "insights-runtime-extractor-tls" not found Sep 08 04:01:13.190699 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:13.190673 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-drx9g_f1262da6-55b5-49dd-8b8b-187c86dc46f6/dns-node-resolver/0.log" Sep 08 04:01:14.391915 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:14.391882 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qcbd8_6698d58b-300e-4f04-a773-1756fcd6f558/node-ca/0.log" Sep 08 04:01:14.821646 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:14.821616 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b5cda57-838f-4ba0-87be-f09808d06d65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kjg2p\" (UID: \"5b5cda57-838f-4ba0-87be-f09808d06d65\") " pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:14.821813 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:14.821669 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b5cda57-838f-4ba0-87be-f09808d06d65-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kjg2p\" (UID: \"5b5cda57-838f-4ba0-87be-f09808d06d65\") " pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:14.821813 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:14.821778 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 04:01:14.821925 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:14.821841 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b5cda57-838f-4ba0-87be-f09808d06d65-insights-runtime-extractor-tls podName:5b5cda57-838f-4ba0-87be-f09808d06d65 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:18.821826903 +0000 UTC m=+52.744287620 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5b5cda57-838f-4ba0-87be-f09808d06d65-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kjg2p" (UID: "5b5cda57-838f-4ba0-87be-f09808d06d65") : secret "insights-runtime-extractor-tls" not found Sep 08 04:01:14.821925 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:14.821845 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 04:01:14.821925 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:14.821916 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b5cda57-838f-4ba0-87be-f09808d06d65-kube-rbac-proxy-cm podName:5b5cda57-838f-4ba0-87be-f09808d06d65 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:18.821896879 +0000 UTC m=+52.744357645 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5b5cda57-838f-4ba0-87be-f09808d06d65-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kjg2p" (UID: "5b5cda57-838f-4ba0-87be-f09808d06d65") : configmap "kube-rbac-proxy" not found Sep 08 04:01:15.191099 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:15.191039 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pv6qv_3e2eec1f-b2a7-43d5-a9e7-f15eafb23937/serve-healthcheck-canary/0.log" Sep 08 04:01:15.791434 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:15.791407 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-xmcbr_56994db0-6aa0-44a7-a540-777271c022ee/kube-storage-version-migrator-operator/0.log" Sep 08 04:01:15.830196 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:15.830167 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57a0643d-28d1-4d11-bb40-1de33627ca6d-metrics-tls\") pod \"dns-default-b8vzf\" (UID: \"57a0643d-28d1-4d11-bb40-1de33627ca6d\") " pod="openshift-dns/dns-default-b8vzf" Sep 08 04:01:15.830315 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:15.830297 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 04:01:15.830374 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:15.830363 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57a0643d-28d1-4d11-bb40-1de33627ca6d-metrics-tls podName:57a0643d-28d1-4d11-bb40-1de33627ca6d nodeName:}" failed. No retries permitted until 2026-09-08 04:01:31.830350393 +0000 UTC m=+65.752811110 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57a0643d-28d1-4d11-bb40-1de33627ca6d-metrics-tls") pod "dns-default-b8vzf" (UID: "57a0643d-28d1-4d11-bb40-1de33627ca6d") : secret "dns-default-metrics-tls" not found Sep 08 04:01:16.598010 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:16.597983 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:17.443495 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:17.443463 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:17.544471 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:17.544444 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:17.648702 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:17.648674 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:17.824380 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:17.824353 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:17.975975 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:17.975950 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:18.540444 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:18.540418 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:18.834612 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:18.834552 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:18.858729 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:18.858704 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b5cda57-838f-4ba0-87be-f09808d06d65-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kjg2p\" (UID: \"5b5cda57-838f-4ba0-87be-f09808d06d65\") " pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:18.858867 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:18.858850 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b5cda57-838f-4ba0-87be-f09808d06d65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kjg2p\" (UID: \"5b5cda57-838f-4ba0-87be-f09808d06d65\") " pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:18.858907 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:18.858868 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 04:01:18.859007 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:18.858995 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b5cda57-838f-4ba0-87be-f09808d06d65-kube-rbac-proxy-cm podName:5b5cda57-838f-4ba0-87be-f09808d06d65 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:26.858975252 +0000 UTC m=+60.781435998 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5b5cda57-838f-4ba0-87be-f09808d06d65-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kjg2p" (UID: "5b5cda57-838f-4ba0-87be-f09808d06d65") : configmap "kube-rbac-proxy" not found Sep 08 04:01:18.859062 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:18.859027 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 04:01:18.859099 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:18.859070 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b5cda57-838f-4ba0-87be-f09808d06d65-insights-runtime-extractor-tls podName:5b5cda57-838f-4ba0-87be-f09808d06d65 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:26.859058445 +0000 UTC m=+60.781519162 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5b5cda57-838f-4ba0-87be-f09808d06d65-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kjg2p" (UID: "5b5cda57-838f-4ba0-87be-f09808d06d65") : secret "insights-runtime-extractor-tls" not found Sep 08 04:01:19.148118 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:19.148053 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:19.349633 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:19.349603 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:19.542686 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:19.542661 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:19.770177 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:19.770149 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b35d6e27-f1ad-4481-a512-36851544fffa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-pqx2g\" (UID: \"b35d6e27-f1ad-4481-a512-36851544fffa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pqx2g" Sep 08 04:01:19.770280 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:19.770183 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a8d25d2-4b61-4484-bcba-d9387ebb641f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-25kxb\" (UID: \"3a8d25d2-4b61-4484-bcba-d9387ebb641f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-25kxb" Sep 08 04:01:19.770322 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:19.770283 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 04:01:19.770373 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:19.770361 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b35d6e27-f1ad-4481-a512-36851544fffa-cluster-monitoring-operator-tls podName:b35d6e27-f1ad-4481-a512-36851544fffa nodeName:}" failed. No retries permitted until 2026-09-08 04:01:35.770343908 +0000 UTC m=+69.692804630 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b35d6e27-f1ad-4481-a512-36851544fffa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-pqx2g" (UID: "b35d6e27-f1ad-4481-a512-36851544fffa") : secret "cluster-monitoring-operator-tls" not found Sep 08 04:01:19.772728 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:19.772697 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a8d25d2-4b61-4484-bcba-d9387ebb641f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-25kxb\" (UID: \"3a8d25d2-4b61-4484-bcba-d9387ebb641f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-25kxb" Sep 08 04:01:19.883299 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:19.883238 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-25kxb" Sep 08 04:01:19.939833 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:19.939798 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:19.971915 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:19.971891 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1a7f679-2737-437e-a718-53fe941b999a-service-ca-bundle\") pod \"router-default-55766f96d-krtn9\" (UID: \"e1a7f679-2737-437e-a718-53fe941b999a\") " pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:19.972031 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:19.971958 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1a7f679-2737-437e-a718-53fe941b999a-metrics-certs\") pod \"router-default-55766f96d-krtn9\" (UID: \"e1a7f679-2737-437e-a718-53fe941b999a\") " pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:19.972031 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:19.971990 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d0fb6dc7-9787-40e0-a9d4-fe3a16f34750-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-6czj4\" (UID: \"d0fb6dc7-9787-40e0-a9d4-fe3a16f34750\") " pod="openshift-network-console/networking-console-plugin-79977f9844-6czj4" Sep 08 04:01:19.972143 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:19.972088 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e1a7f679-2737-437e-a718-53fe941b999a-service-ca-bundle podName:e1a7f679-2737-437e-a718-53fe941b999a nodeName:}" failed. No retries permitted until 2026-09-08 04:01:35.972064961 +0000 UTC m=+69.894525697 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e1a7f679-2737-437e-a718-53fe941b999a-service-ca-bundle") pod "router-default-55766f96d-krtn9" (UID: "e1a7f679-2737-437e-a718-53fe941b999a") : configmap references non-existent config key: service-ca.crt Sep 08 04:01:19.972143 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:19.972100 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 08 04:01:19.972227 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:19.972149 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a7f679-2737-437e-a718-53fe941b999a-metrics-certs podName:e1a7f679-2737-437e-a718-53fe941b999a nodeName:}" failed. No retries permitted until 2026-09-08 04:01:35.972135199 +0000 UTC m=+69.894595958 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1a7f679-2737-437e-a718-53fe941b999a-metrics-certs") pod "router-default-55766f96d-krtn9" (UID: "e1a7f679-2737-437e-a718-53fe941b999a") : secret "router-metrics-certs-default" not found Sep 08 04:01:19.972227 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:19.972090 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 08 04:01:19.972227 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:19.972179 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0fb6dc7-9787-40e0-a9d4-fe3a16f34750-networking-console-plugin-cert podName:d0fb6dc7-9787-40e0-a9d4-fe3a16f34750 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:35.972172807 +0000 UTC m=+69.894633585 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d0fb6dc7-9787-40e0-a9d4-fe3a16f34750-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-6czj4" (UID: "d0fb6dc7-9787-40e0-a9d4-fe3a16f34750") : secret "networking-console-plugin-cert" not found Sep 08 04:01:20.073034 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:20.073009 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-tls\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:20.073170 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:20.073152 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 04:01:20.073237 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:20.073173 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db884754d-bntvx: secret "image-registry-tls" not found Sep 08 04:01:20.073237 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:20.073223 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-tls podName:b0ce25c1-3ba6-4f70-9457-7ded8bc28360 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:36.073208156 +0000 UTC m=+69.995668899 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-tls") pod "image-registry-7db884754d-bntvx" (UID: "b0ce25c1-3ba6-4f70-9457-7ded8bc28360") : secret "image-registry-tls" not found Sep 08 04:01:20.160371 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:20.160314 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-25kxb"] Sep 08 04:01:20.232426 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:20.232401 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:20.294346 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:20.294323 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:20.577345 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:20.577318 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:20.765336 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:20.765299 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-25kxb" event={"ID":"3a8d25d2-4b61-4484-bcba-d9387ebb641f","Type":"ContainerStarted","Data":"51322e26f6191c94b369770ef8677f3bdfced763c873239fc0c688025d1c21b1"} Sep 08 04:01:20.860461 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:20.860379 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:20.961088 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:20.961066 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:21.560579 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:21.560546 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:21.660942 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:21.660908 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:21.763328 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:21.763294 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:22.537425 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:22.537397 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:22.772280 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:22.772245 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-25kxb" event={"ID":"3a8d25d2-4b61-4484-bcba-d9387ebb641f","Type":"ContainerStarted","Data":"9262b073eab3ce024664f4080eaf838824e2106014ddbb06c7578b5133415e57"} Sep 08 04:01:22.772675 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:22.772307 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-25kxb" event={"ID":"3a8d25d2-4b61-4484-bcba-d9387ebb641f","Type":"ContainerStarted","Data":"b284bcf4ecd2d3c53c87870c1d5378b6849646f386e3f66e839ca197da7ad43a"} Sep 08 04:01:22.789981 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:22.789904 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-25kxb" podStartSLOduration=18.162661734 podStartE2EDuration="19.789888992s" totalImagesPullingTime="1.627227258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:01:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:22.789062448 +0000 UTC m=+56.711523205" watchObservedRunningTime="2026-09-08 04:01:22.789888992 +0000 UTC m=+56.712349730" Sep 08 04:01:22.938576 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:22.938548 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:23.151146 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:23.151080 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:23.550490 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:23.550464 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:23.727977 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:23.727953 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:24.150798 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:24.150737 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:24.743194 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:24.743158 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:24.942075 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:24.942049 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:25.340152 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:25.340127 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:25.537910 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:25.537884 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:25.697331 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:25.697277 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-86px5" Sep 08 04:01:26.139760 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:26.139736 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:26.297428 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:26.297394 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:26.740543 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:26.740488 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:26.933195 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:26.933145 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b5cda57-838f-4ba0-87be-f09808d06d65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kjg2p\" (UID: \"5b5cda57-838f-4ba0-87be-f09808d06d65\") " pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:26.933330 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:26.933222 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b5cda57-838f-4ba0-87be-f09808d06d65-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kjg2p\" (UID: \"5b5cda57-838f-4ba0-87be-f09808d06d65\") " pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:26.933467 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:26.933452 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 04:01:26.933555 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:26.933543 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b5cda57-838f-4ba0-87be-f09808d06d65-insights-runtime-extractor-tls podName:5b5cda57-838f-4ba0-87be-f09808d06d65 nodeName:}" failed. No retries permitted until 2026-09-08 04:01:42.933526303 +0000 UTC m=+76.855987021 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5b5cda57-838f-4ba0-87be-f09808d06d65-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kjg2p" (UID: "5b5cda57-838f-4ba0-87be-f09808d06d65") : secret "insights-runtime-extractor-tls" not found Sep 08 04:01:26.933758 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:26.933734 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b5cda57-838f-4ba0-87be-f09808d06d65-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kjg2p\" (UID: \"5b5cda57-838f-4ba0-87be-f09808d06d65\") " pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:27.032283 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:27.032253 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:27.143915 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:27.143883 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:27.339387 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:27.339334 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:27.944570 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:27.944543 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:28.135885 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:28.135854 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:31.875968 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:31.875934 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57a0643d-28d1-4d11-bb40-1de33627ca6d-metrics-tls\") pod \"dns-default-b8vzf\" (UID: \"57a0643d-28d1-4d11-bb40-1de33627ca6d\") " pod="openshift-dns/dns-default-b8vzf" Sep 08 04:01:31.878423 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:31.878397 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57a0643d-28d1-4d11-bb40-1de33627ca6d-metrics-tls\") pod \"dns-default-b8vzf\" (UID: \"57a0643d-28d1-4d11-bb40-1de33627ca6d\") " pod="openshift-dns/dns-default-b8vzf" Sep 08 04:01:32.059209 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:32.059182 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-26sdq\"" Sep 08 04:01:32.067228 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:32.067209 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b8vzf" Sep 08 04:01:32.279118 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:32.279091 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff12a182-ed89-4513-938b-69971199ad25-metrics-certs\") pod \"network-metrics-daemon-rzwvc\" (UID: \"ff12a182-ed89-4513-938b-69971199ad25\") " pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:01:32.282198 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:32.282179 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 04:01:32.292223 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:32.292198 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff12a182-ed89-4513-938b-69971199ad25-metrics-certs\") pod \"network-metrics-daemon-rzwvc\" (UID: \"ff12a182-ed89-4513-938b-69971199ad25\") " pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:01:32.343641 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:32.343624 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b8vzf"] Sep 08 04:01:32.346068 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:01:32.346042 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57a0643d_28d1_4d11_bb40_1de33627ca6d.slice/crio-6215389d7e5e57486a646a2d80c9e05bf8733897228bebcadbf876d34d1a7d39 WatchSource:0}: Error finding container 6215389d7e5e57486a646a2d80c9e05bf8733897228bebcadbf876d34d1a7d39: Status 404 returned error can't find the container with id 6215389d7e5e57486a646a2d80c9e05bf8733897228bebcadbf876d34d1a7d39 Sep 08 04:01:32.379747 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:32.379722 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-original-pull-secret\") pod \"global-pull-secret-syncer-x4dkl\" (UID: \"c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb\") " pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:01:32.382367 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:32.382352 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 04:01:32.392112 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:32.392097 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb-original-pull-secret\") pod \"global-pull-secret-syncer-x4dkl\" (UID: \"c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb\") " pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:01:32.480073 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:32.480043 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4dkl" Sep 08 04:01:32.480222 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:32.480203 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tns48\" (UniqueName: \"kubernetes.io/projected/4c61ba14-8d9f-4634-9a0c-aac888042abe-kube-api-access-tns48\") pod \"network-check-target-l6l2z\" (UID: \"4c61ba14-8d9f-4634-9a0c-aac888042abe\") " pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:01:32.483002 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:32.482987 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 08 04:01:32.486166 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:32.486147 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-blqwr\"" Sep 08 04:01:32.494099 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:32.494083 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzwvc" Sep 08 04:01:32.494682 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:32.494478 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 08 04:01:32.504216 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:32.504191 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tns48\" (UniqueName: \"kubernetes.io/projected/4c61ba14-8d9f-4634-9a0c-aac888042abe-kube-api-access-tns48\") pod \"network-check-target-l6l2z\" (UID: \"4c61ba14-8d9f-4634-9a0c-aac888042abe\") " pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:01:32.750393 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:32.750371 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x4dkl"] Sep 08 04:01:32.752613 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:01:32.752585 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc075d89b_4d37_4e76_a9e5_d61c6e9cf2eb.slice/crio-9b8c85f4bafdd0997f8d1012ffb45a800a50f1ea69d3ce092b7d62a4eb7f42e7 WatchSource:0}: Error finding container 9b8c85f4bafdd0997f8d1012ffb45a800a50f1ea69d3ce092b7d62a4eb7f42e7: Status 404 returned error can't find the container with id 9b8c85f4bafdd0997f8d1012ffb45a800a50f1ea69d3ce092b7d62a4eb7f42e7 Sep 08 04:01:32.772830 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:32.772813 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rzwvc"] Sep 08 04:01:32.774645 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:01:32.774621 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff12a182_ed89_4513_938b_69971199ad25.slice/crio-42b5747b6a744c5af425a67f89668342a6768462e55d1cde69797562a104c216 WatchSource:0}: Error finding container 42b5747b6a744c5af425a67f89668342a6768462e55d1cde69797562a104c216: Status 404 returned error can't find the container with id 42b5747b6a744c5af425a67f89668342a6768462e55d1cde69797562a104c216 Sep 08 04:01:32.776338 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:32.776308 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-f9485\"" Sep 08 04:01:32.784327 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:32.784281 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:01:32.797126 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:32.797104 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x4dkl" event={"ID":"c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb","Type":"ContainerStarted","Data":"9b8c85f4bafdd0997f8d1012ffb45a800a50f1ea69d3ce092b7d62a4eb7f42e7"} Sep 08 04:01:32.798443 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:32.798415 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rzwvc" event={"ID":"ff12a182-ed89-4513-938b-69971199ad25","Type":"ContainerStarted","Data":"42b5747b6a744c5af425a67f89668342a6768462e55d1cde69797562a104c216"} Sep 08 04:01:32.799543 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:32.799522 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b8vzf" event={"ID":"57a0643d-28d1-4d11-bb40-1de33627ca6d","Type":"ContainerStarted","Data":"6215389d7e5e57486a646a2d80c9e05bf8733897228bebcadbf876d34d1a7d39"} Sep 08 04:01:33.054229 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:33.054162 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l6l2z"] Sep 08 04:01:33.056989 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:01:33.056958 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c61ba14_8d9f_4634_9a0c_aac888042abe.slice/crio-a861cd4141eef1a2850c37a5e97843867377e85966982e001bd580caba33e7ae WatchSource:0}: Error finding container a861cd4141eef1a2850c37a5e97843867377e85966982e001bd580caba33e7ae: Status 404 returned error can't find the container with id a861cd4141eef1a2850c37a5e97843867377e85966982e001bd580caba33e7ae Sep 08 04:01:33.804807 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:33.804772 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l6l2z" event={"ID":"4c61ba14-8d9f-4634-9a0c-aac888042abe","Type":"ContainerStarted","Data":"a861cd4141eef1a2850c37a5e97843867377e85966982e001bd580caba33e7ae"} Sep 08 04:01:34.812428 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:34.811922 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rzwvc" event={"ID":"ff12a182-ed89-4513-938b-69971199ad25","Type":"ContainerStarted","Data":"e137737362c64e5b01762907e1cce11ec0b89a3f5b947b8be5a08cf75d625fd2"} Sep 08 04:01:34.812428 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:34.811977 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rzwvc" event={"ID":"ff12a182-ed89-4513-938b-69971199ad25","Type":"ContainerStarted","Data":"274b72e240755952e94aeaa1fa36adaef4df23c298b831d602b7ef59d4053fc0"} Sep 08 04:01:34.815005 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:34.814926 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b8vzf" event={"ID":"57a0643d-28d1-4d11-bb40-1de33627ca6d","Type":"ContainerStarted","Data":"39f194251e0682118dbcbe8319098199f6bbd6068e01879f538c7f805b9c16ad"} Sep 08 04:01:34.815005 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:34.814972 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b8vzf" event={"ID":"57a0643d-28d1-4d11-bb40-1de33627ca6d","Type":"ContainerStarted","Data":"383aedae65a074ab1333618de61ac358529567c1c9bc2e3e5ca3a04006ce5aeb"} Sep 08 04:01:34.815204 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:34.815122 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-b8vzf" Sep 08 04:01:34.828771 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:34.828438 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rzwvc" podStartSLOduration=66.260506154 podStartE2EDuration="1m7.828420551s" totalImagesPullingTime="1.567914397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:34.826990934 +0000 UTC m=+68.749451703" watchObservedRunningTime="2026-09-08 04:01:34.828420551 +0000 UTC m=+68.750881291" Sep 08 04:01:34.844403 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:34.844298 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-b8vzf" podStartSLOduration=33.843492655 podStartE2EDuration="35.844281379s" totalImagesPullingTime="2.000788724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:00:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:34.843241039 +0000 UTC m=+68.765701778" watchObservedRunningTime="2026-09-08 04:01:34.844281379 +0000 UTC m=+68.766742114" Sep 08 04:01:35.779764 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:35.779730 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7db884754d-bntvx"] Sep 08 04:01:35.779972 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:01:35.779946 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7db884754d-bntvx" podUID="b0ce25c1-3ba6-4f70-9457-7ded8bc28360" Sep 08 04:01:35.810681 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:35.810639 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b35d6e27-f1ad-4481-a512-36851544fffa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-pqx2g\" (UID: \"b35d6e27-f1ad-4481-a512-36851544fffa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pqx2g" Sep 08 04:01:35.814142 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:35.814029 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-657cf94686-prdgx"] Sep 08 04:01:35.815141 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:35.815116 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b35d6e27-f1ad-4481-a512-36851544fffa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-pqx2g\" (UID: \"b35d6e27-f1ad-4481-a512-36851544fffa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pqx2g" Sep 08 04:01:35.817260 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:35.816942 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657cf94686-prdgx" Sep 08 04:01:35.819131 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:35.819095 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:35.820023 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:35.819925 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 08 04:01:35.820134 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:35.820075 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 08 04:01:35.820543 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:35.820522 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 08 04:01:35.822214 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:35.822193 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 08 04:01:35.833950 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:35.833929 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-657cf94686-prdgx"] Sep 08 04:01:35.837518 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:35.837485 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:35.868219 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:35.868193 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:35.896204 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:35.895679 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64d47bbb76-26f2f"] Sep 08 04:01:35.898166 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:35.898059 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:35.922889 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:35.922868 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64d47bbb76-26f2f"] Sep 08 04:01:35.967638 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:35.967605 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:36.013337 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.013307 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-ca-trust-extracted\") pod \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " Sep 08 04:01:36.013464 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.013353 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-certificates\") pod \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " Sep 08 04:01:36.013464 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.013382 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-trusted-ca\") pod \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " Sep 08 04:01:36.013464 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.013406 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-bound-sa-token\") pod \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " Sep 08 04:01:36.013464 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.013431 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-image-registry-private-configuration\") pod \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " Sep 08 04:01:36.013695 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.013464 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-installation-pull-secrets\") pod \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " Sep 08 04:01:36.013750 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.013713 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58859\" (UniqueName: \"kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-kube-api-access-58859\") pod \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " Sep 08 04:01:36.013835 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.013818 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2cf49095-f078-4f11-a993-697cebff6172-registry-certificates\") pod \"image-registry-64d47bbb76-26f2f\" (UID: \"2cf49095-f078-4f11-a993-697cebff6172\") " pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:36.013835 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.013764 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b0ce25c1-3ba6-4f70-9457-7ded8bc28360" (UID: "b0ce25c1-3ba6-4f70-9457-7ded8bc28360"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:01:36.013945 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.013858 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2cf49095-f078-4f11-a993-697cebff6172-image-registry-private-configuration\") pod \"image-registry-64d47bbb76-26f2f\" (UID: \"2cf49095-f078-4f11-a993-697cebff6172\") " pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:36.013945 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.013898 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cf49095-f078-4f11-a993-697cebff6172-bound-sa-token\") pod \"image-registry-64d47bbb76-26f2f\" (UID: \"2cf49095-f078-4f11-a993-697cebff6172\") " pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:36.014039 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.013932 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b0ce25c1-3ba6-4f70-9457-7ded8bc28360" (UID: "b0ce25c1-3ba6-4f70-9457-7ded8bc28360"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:01:36.014039 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.013954 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b0ce25c1-3ba6-4f70-9457-7ded8bc28360" (UID: "b0ce25c1-3ba6-4f70-9457-7ded8bc28360"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:01:36.014039 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.013988 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1a7f679-2737-437e-a718-53fe941b999a-service-ca-bundle\") pod \"router-default-55766f96d-krtn9\" (UID: \"e1a7f679-2737-437e-a718-53fe941b999a\") " pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:36.014178 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.014071 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c6629db6-b1af-479a-ad73-d44aba8b624f-klusterlet-config\") pod \"klusterlet-addon-workmgr-657cf94686-prdgx\" (UID: \"c6629db6-b1af-479a-ad73-d44aba8b624f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657cf94686-prdgx" Sep 08 04:01:36.014178 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.014104 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-664pk\" (UniqueName: \"kubernetes.io/projected/c6629db6-b1af-479a-ad73-d44aba8b624f-kube-api-access-664pk\") pod \"klusterlet-addon-workmgr-657cf94686-prdgx\" (UID: \"c6629db6-b1af-479a-ad73-d44aba8b624f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657cf94686-prdgx" Sep 08 04:01:36.014178 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.014132 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2cf49095-f078-4f11-a993-697cebff6172-trusted-ca\") pod \"image-registry-64d47bbb76-26f2f\" (UID: \"2cf49095-f078-4f11-a993-697cebff6172\") " pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:36.014316 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.014185 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2cf49095-f078-4f11-a993-697cebff6172-installation-pull-secrets\") pod \"image-registry-64d47bbb76-26f2f\" (UID: \"2cf49095-f078-4f11-a993-697cebff6172\") " pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:36.014997 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.014673 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1a7f679-2737-437e-a718-53fe941b999a-metrics-certs\") pod \"router-default-55766f96d-krtn9\" (UID: \"e1a7f679-2737-437e-a718-53fe941b999a\") " pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:36.014997 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.014710 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1a7f679-2737-437e-a718-53fe941b999a-service-ca-bundle\") pod \"router-default-55766f96d-krtn9\" (UID: \"e1a7f679-2737-437e-a718-53fe941b999a\") " pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:36.014997 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.014832 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cf49095-f078-4f11-a993-697cebff6172-registry-tls\") pod \"image-registry-64d47bbb76-26f2f\" (UID: \"2cf49095-f078-4f11-a993-697cebff6172\") " pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:36.014997 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.014861 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdvx6\" (UniqueName: \"kubernetes.io/projected/2cf49095-f078-4f11-a993-697cebff6172-kube-api-access-cdvx6\") pod \"image-registry-64d47bbb76-26f2f\" (UID: \"2cf49095-f078-4f11-a993-697cebff6172\") " pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:36.014997 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.014961 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6629db6-b1af-479a-ad73-d44aba8b624f-tmp\") pod \"klusterlet-addon-workmgr-657cf94686-prdgx\" (UID: \"c6629db6-b1af-479a-ad73-d44aba8b624f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657cf94686-prdgx" Sep 08 04:01:36.015294 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.015019 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d0fb6dc7-9787-40e0-a9d4-fe3a16f34750-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-6czj4\" (UID: \"d0fb6dc7-9787-40e0-a9d4-fe3a16f34750\") " pod="openshift-network-console/networking-console-plugin-79977f9844-6czj4" Sep 08 04:01:36.015294 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.015052 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2cf49095-f078-4f11-a993-697cebff6172-ca-trust-extracted\") pod \"image-registry-64d47bbb76-26f2f\" (UID: \"2cf49095-f078-4f11-a993-697cebff6172\") " pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:36.015294 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.015099 2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-ca-trust-extracted\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:01:36.015294 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.015114 2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-certificates\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:01:36.015294 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.015128 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-trusted-ca\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:01:36.017898 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.017879 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1a7f679-2737-437e-a718-53fe941b999a-metrics-certs\") pod \"router-default-55766f96d-krtn9\" (UID: \"e1a7f679-2737-437e-a718-53fe941b999a\") " pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:36.017985 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.017900 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d0fb6dc7-9787-40e0-a9d4-fe3a16f34750-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-6czj4\" (UID: \"d0fb6dc7-9787-40e0-a9d4-fe3a16f34750\") " pod="openshift-network-console/networking-console-plugin-79977f9844-6czj4" Sep 08 04:01:36.018077 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.018030 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b0ce25c1-3ba6-4f70-9457-7ded8bc28360" (UID: "b0ce25c1-3ba6-4f70-9457-7ded8bc28360"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 04:01:36.018409 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.018381 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b0ce25c1-3ba6-4f70-9457-7ded8bc28360" (UID: "b0ce25c1-3ba6-4f70-9457-7ded8bc28360"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 04:01:36.018409 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.018441 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-kube-api-access-58859" (OuterVolumeSpecName: "kube-api-access-58859") pod "b0ce25c1-3ba6-4f70-9457-7ded8bc28360" (UID: "b0ce25c1-3ba6-4f70-9457-7ded8bc28360"). InnerVolumeSpecName "kube-api-access-58859". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:01:36.018409 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.018484 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b0ce25c1-3ba6-4f70-9457-7ded8bc28360" (UID: "b0ce25c1-3ba6-4f70-9457-7ded8bc28360"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:01:36.031588 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.031540 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4b7vt\"" Sep 08 04:01:36.039312 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.039293 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:36.069135 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.069107 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:36.079727 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.079701 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-k54h4\"" Sep 08 04:01:36.087109 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.086960 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pqx2g" Sep 08 04:01:36.116144 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.116117 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2cf49095-f078-4f11-a993-697cebff6172-registry-certificates\") pod \"image-registry-64d47bbb76-26f2f\" (UID: \"2cf49095-f078-4f11-a993-697cebff6172\") " pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:36.116231 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.116162 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2cf49095-f078-4f11-a993-697cebff6172-image-registry-private-configuration\") pod \"image-registry-64d47bbb76-26f2f\" (UID: \"2cf49095-f078-4f11-a993-697cebff6172\") " pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:36.116394 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.116365 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cf49095-f078-4f11-a993-697cebff6172-bound-sa-token\") pod \"image-registry-64d47bbb76-26f2f\" (UID: \"2cf49095-f078-4f11-a993-697cebff6172\") " pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:36.116533 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.116486 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c6629db6-b1af-479a-ad73-d44aba8b624f-klusterlet-config\") pod \"klusterlet-addon-workmgr-657cf94686-prdgx\" (UID: \"c6629db6-b1af-479a-ad73-d44aba8b624f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657cf94686-prdgx" Sep 08 04:01:36.116647 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.116538 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-664pk\" (UniqueName: \"kubernetes.io/projected/c6629db6-b1af-479a-ad73-d44aba8b624f-kube-api-access-664pk\") pod \"klusterlet-addon-workmgr-657cf94686-prdgx\" (UID: \"c6629db6-b1af-479a-ad73-d44aba8b624f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657cf94686-prdgx" Sep 08 04:01:36.116647 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.116564 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2cf49095-f078-4f11-a993-697cebff6172-trusted-ca\") pod \"image-registry-64d47bbb76-26f2f\" (UID: \"2cf49095-f078-4f11-a993-697cebff6172\") " pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:36.116647 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.116601 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2cf49095-f078-4f11-a993-697cebff6172-installation-pull-secrets\") pod \"image-registry-64d47bbb76-26f2f\" (UID: \"2cf49095-f078-4f11-a993-697cebff6172\") " pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:36.116647 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.116637 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-tls\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:36.116842 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.116660 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cf49095-f078-4f11-a993-697cebff6172-registry-tls\") pod \"image-registry-64d47bbb76-26f2f\" (UID: \"2cf49095-f078-4f11-a993-697cebff6172\") " pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:36.116842 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.116683 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdvx6\" (UniqueName: \"kubernetes.io/projected/2cf49095-f078-4f11-a993-697cebff6172-kube-api-access-cdvx6\") pod \"image-registry-64d47bbb76-26f2f\" (UID: \"2cf49095-f078-4f11-a993-697cebff6172\") " pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:36.117054 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.117000 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2cf49095-f078-4f11-a993-697cebff6172-registry-certificates\") pod \"image-registry-64d47bbb76-26f2f\" (UID: \"2cf49095-f078-4f11-a993-697cebff6172\") " pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:36.117054 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.117020 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6629db6-b1af-479a-ad73-d44aba8b624f-tmp\") pod \"klusterlet-addon-workmgr-657cf94686-prdgx\" (UID: \"c6629db6-b1af-479a-ad73-d44aba8b624f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657cf94686-prdgx" Sep 08 04:01:36.117209 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.117066 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2cf49095-f078-4f11-a993-697cebff6172-ca-trust-extracted\") pod \"image-registry-64d47bbb76-26f2f\" (UID: \"2cf49095-f078-4f11-a993-697cebff6172\") " pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:36.117209 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.117111 2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-installation-pull-secrets\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:01:36.117209 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.117129 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58859\" (UniqueName: \"kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-kube-api-access-58859\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:01:36.117209 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.117144 2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-bound-sa-token\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:01:36.117209 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.117158 2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-image-registry-private-configuration\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:01:36.117838 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.117450 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2cf49095-f078-4f11-a993-697cebff6172-trusted-ca\") pod \"image-registry-64d47bbb76-26f2f\" (UID: \"2cf49095-f078-4f11-a993-697cebff6172\") " pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:36.117838 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.117456 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2cf49095-f078-4f11-a993-697cebff6172-ca-trust-extracted\") pod \"image-registry-64d47bbb76-26f2f\" (UID: \"2cf49095-f078-4f11-a993-697cebff6172\") " pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:36.117838 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.117799 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6629db6-b1af-479a-ad73-d44aba8b624f-tmp\") pod \"klusterlet-addon-workmgr-657cf94686-prdgx\" (UID: \"c6629db6-b1af-479a-ad73-d44aba8b624f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657cf94686-prdgx" Sep 08 04:01:36.119191 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.119167 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2cf49095-f078-4f11-a993-697cebff6172-image-registry-private-configuration\") pod \"image-registry-64d47bbb76-26f2f\" (UID: \"2cf49095-f078-4f11-a993-697cebff6172\") " pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:36.119427 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.119367 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2cf49095-f078-4f11-a993-697cebff6172-installation-pull-secrets\") pod \"image-registry-64d47bbb76-26f2f\" (UID: \"2cf49095-f078-4f11-a993-697cebff6172\") " pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:36.119530 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.119471 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-tls\") pod \"image-registry-7db884754d-bntvx\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:36.119593 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.119546 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cf49095-f078-4f11-a993-697cebff6172-registry-tls\") pod \"image-registry-64d47bbb76-26f2f\" (UID: \"2cf49095-f078-4f11-a993-697cebff6172\") " pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:36.120172 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.120146 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c6629db6-b1af-479a-ad73-d44aba8b624f-klusterlet-config\") pod \"klusterlet-addon-workmgr-657cf94686-prdgx\" (UID: \"c6629db6-b1af-479a-ad73-d44aba8b624f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657cf94686-prdgx" Sep 08 04:01:36.125437 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.125393 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdvx6\" (UniqueName: \"kubernetes.io/projected/2cf49095-f078-4f11-a993-697cebff6172-kube-api-access-cdvx6\") pod \"image-registry-64d47bbb76-26f2f\" (UID: \"2cf49095-f078-4f11-a993-697cebff6172\") " pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:36.125555 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.125453 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cf49095-f078-4f11-a993-697cebff6172-bound-sa-token\") pod \"image-registry-64d47bbb76-26f2f\" (UID: \"2cf49095-f078-4f11-a993-697cebff6172\") " pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:36.125555 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.125471 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-664pk\" (UniqueName: \"kubernetes.io/projected/c6629db6-b1af-479a-ad73-d44aba8b624f-kube-api-access-664pk\") pod \"klusterlet-addon-workmgr-657cf94686-prdgx\" (UID: \"c6629db6-b1af-479a-ad73-d44aba8b624f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657cf94686-prdgx" Sep 08 04:01:36.135912 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.135892 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657cf94686-prdgx" Sep 08 04:01:36.166885 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.166858 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:36.210977 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.210953 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qx7pf\"" Sep 08 04:01:36.218402 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.218378 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-tls\") pod \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\" (UID: \"b0ce25c1-3ba6-4f70-9457-7ded8bc28360\") " Sep 08 04:01:36.219015 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.218994 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:36.220412 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.220390 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b0ce25c1-3ba6-4f70-9457-7ded8bc28360" (UID: "b0ce25c1-3ba6-4f70-9457-7ded8bc28360"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:01:36.306685 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.306630 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9wm7x\"" Sep 08 04:01:36.314645 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.314621 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-6czj4" Sep 08 04:01:36.318846 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.318823 2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b0ce25c1-3ba6-4f70-9457-7ded8bc28360-registry-tls\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:01:36.712029 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.711995 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:36.777429 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.777401 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:36.822019 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.821961 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7db884754d-bntvx" Sep 08 04:01:36.851669 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.851646 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7db884754d-bntvx"] Sep 08 04:01:36.911226 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:36.911194 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7db884754d-bntvx"] Sep 08 04:01:38.246757 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:38.246696 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:38.312201 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:38.311077 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-6czj4"] Sep 08 04:01:38.313063 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:38.312819 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64d47bbb76-26f2f"] Sep 08 04:01:38.313809 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:38.313626 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:38.314276 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:38.314149 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-55766f96d-krtn9"] Sep 08 04:01:38.314343 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:01:38.314271 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0fb6dc7_9787_40e0_a9d4_fe3a16f34750.slice/crio-e3bdbc8429b4e837e77254d56b8b9e17f3e8b027ecb68d5439ef98b4cef9423f WatchSource:0}: Error finding container e3bdbc8429b4e837e77254d56b8b9e17f3e8b027ecb68d5439ef98b4cef9423f: Status 404 returned error can't find the container with id e3bdbc8429b4e837e77254d56b8b9e17f3e8b027ecb68d5439ef98b4cef9423f Sep 08 04:01:38.316626 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:01:38.316599 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf49095_f078_4f11_a993_697cebff6172.slice/crio-aec12c2fbe7914de0de963b27dd44f1a6ff6987b8af8971c2e26b7c7843036ff WatchSource:0}: Error finding container aec12c2fbe7914de0de963b27dd44f1a6ff6987b8af8971c2e26b7c7843036ff: Status 404 returned error can't find the container with id aec12c2fbe7914de0de963b27dd44f1a6ff6987b8af8971c2e26b7c7843036ff Sep 08 04:01:38.317626 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:01:38.317603 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a7f679_2737_437e_a718_53fe941b999a.slice/crio-def7368be6318a73c542e1ee0d2488fb6e6b2fd7a674d100d0b00a2e357b9f3d WatchSource:0}: Error finding container def7368be6318a73c542e1ee0d2488fb6e6b2fd7a674d100d0b00a2e357b9f3d: Status 404 returned error can't find the container with id def7368be6318a73c542e1ee0d2488fb6e6b2fd7a674d100d0b00a2e357b9f3d Sep 08 04:01:38.321727 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:38.321704 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-657cf94686-prdgx"] Sep 08 04:01:38.324571 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:38.324549 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pqx2g"] Sep 08 04:01:38.324732 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:01:38.324642 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6629db6_b1af_479a_ad73_d44aba8b624f.slice/crio-0ba42c3b95120c28bc1a502a38b7dc3b159a19caf2b309ec8958f8ba6e7b5614 WatchSource:0}: Error finding container 0ba42c3b95120c28bc1a502a38b7dc3b159a19caf2b309ec8958f8ba6e7b5614: Status 404 returned error can't find the container with id 0ba42c3b95120c28bc1a502a38b7dc3b159a19caf2b309ec8958f8ba6e7b5614 Sep 08 04:01:38.327119 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:01:38.327102 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb35d6e27_f1ad_4481_a512_36851544fffa.slice/crio-adc2aa5a9933940d4aabf611b6be4dd493927d2cb636d43b677cf6c93c1ecbf5 WatchSource:0}: Error finding container adc2aa5a9933940d4aabf611b6be4dd493927d2cb636d43b677cf6c93c1ecbf5: Status 404 returned error can't find the container with id adc2aa5a9933940d4aabf611b6be4dd493927d2cb636d43b677cf6c93c1ecbf5 Sep 08 04:01:38.567325 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:38.567291 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0ce25c1-3ba6-4f70-9457-7ded8bc28360" path="/var/lib/kubelet/pods/b0ce25c1-3ba6-4f70-9457-7ded8bc28360/volumes" Sep 08 04:01:38.830159 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:38.830073 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657cf94686-prdgx" event={"ID":"c6629db6-b1af-479a-ad73-d44aba8b624f","Type":"ContainerStarted","Data":"0ba42c3b95120c28bc1a502a38b7dc3b159a19caf2b309ec8958f8ba6e7b5614"} Sep 08 04:01:38.831751 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:38.831718 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l6l2z" event={"ID":"4c61ba14-8d9f-4634-9a0c-aac888042abe","Type":"ContainerStarted","Data":"f12ccc876c9fa16c13a9e1064609e81e711b214d49362af1e1b12135f750497e"} Sep 08 04:01:38.831883 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:38.831822 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:01:38.833441 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:38.833417 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" event={"ID":"2cf49095-f078-4f11-a993-697cebff6172","Type":"ContainerStarted","Data":"81c653e31b4cbe49bd14cc6855d74fc0ef24347d75d75384415f3fa1f412dbb8"} Sep 08 04:01:38.833561 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:38.833463 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:38.833561 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:38.833484 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" event={"ID":"2cf49095-f078-4f11-a993-697cebff6172","Type":"ContainerStarted","Data":"aec12c2fbe7914de0de963b27dd44f1a6ff6987b8af8971c2e26b7c7843036ff"} Sep 08 04:01:38.835045 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:38.835017 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x4dkl" event={"ID":"c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb","Type":"ContainerStarted","Data":"5f0833741212e8b0f35e83fed49baeefb22fd6dc2d393945fd61b185e1c34ba0"} Sep 08 04:01:38.836216 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:38.836195 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pqx2g" event={"ID":"b35d6e27-f1ad-4481-a512-36851544fffa","Type":"ContainerStarted","Data":"adc2aa5a9933940d4aabf611b6be4dd493927d2cb636d43b677cf6c93c1ecbf5"} Sep 08 04:01:38.837830 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:38.837812 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-55766f96d-krtn9" event={"ID":"e1a7f679-2737-437e-a718-53fe941b999a","Type":"ContainerStarted","Data":"68db58d8b623b38c1b39d1adb9c14978db375299954f09404577909dcd164e5c"} Sep 08 04:01:38.837903 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:38.837842 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-55766f96d-krtn9" event={"ID":"e1a7f679-2737-437e-a718-53fe941b999a","Type":"ContainerStarted","Data":"def7368be6318a73c542e1ee0d2488fb6e6b2fd7a674d100d0b00a2e357b9f3d"} Sep 08 04:01:38.838907 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:38.838890 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-6czj4" event={"ID":"d0fb6dc7-9787-40e0-a9d4-fe3a16f34750","Type":"ContainerStarted","Data":"e3bdbc8429b4e837e77254d56b8b9e17f3e8b027ecb68d5439ef98b4cef9423f"} Sep 08 04:01:38.850679 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:38.850643 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-l6l2z" podStartSLOduration=66.906243314 podStartE2EDuration="1m11.850631521s" totalImagesPullingTime="4.944388207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:38.850169355 +0000 UTC m=+72.772630118" watchObservedRunningTime="2026-09-08 04:01:38.850631521 +0000 UTC m=+72.773092264" Sep 08 04:01:38.867159 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:38.867120 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-x4dkl" podStartSLOduration=66.625887772 podStartE2EDuration="1m11.867107705s" totalImagesPullingTime="5.241219933s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 04:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:38.86573749 +0000 UTC m=+72.788198258" watchObservedRunningTime="2026-09-08 04:01:38.867107705 +0000 UTC m=+72.789568443" Sep 08 04:01:38.886459 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:38.886424 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-55766f96d-krtn9" podStartSLOduration=34.886413279 podStartE2EDuration="34.886413279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:38.885573729 +0000 UTC m=+72.808034471" watchObservedRunningTime="2026-09-08 04:01:38.886413279 +0000 UTC m=+72.808874018" Sep 08 04:01:38.907486 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:38.907454 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" podStartSLOduration=3.9074442080000003 podStartE2EDuration="3.907444208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:38.906253525 +0000 UTC m=+72.828714287" watchObservedRunningTime="2026-09-08 04:01:38.907444208 +0000 UTC m=+72.829904938" Sep 08 04:01:39.039765 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:39.039710 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:39.042527 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:39.042489 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:39.844592 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:39.844559 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:39.845897 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:39.845669 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-55766f96d-krtn9" Sep 08 04:01:40.994441 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:40.994398 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:41.070227 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:41.070166 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:41.269161 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:41.269132 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:42.590692 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:42.590642 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:42.857129 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:42.857044 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pqx2g" event={"ID":"b35d6e27-f1ad-4481-a512-36851544fffa","Type":"ContainerStarted","Data":"bca9d6602bfd0ca7e263e9bde773ce283c640a65ed9e6abe68b869909756cfb4"} Sep 08 04:01:42.858805 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:42.858774 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-6czj4" event={"ID":"d0fb6dc7-9787-40e0-a9d4-fe3a16f34750","Type":"ContainerStarted","Data":"8798a6d9008b35b5422db831094e8817b17c037ec3a8e43b0a0603838f4b6232"} Sep 08 04:01:42.860457 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:42.860434 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657cf94686-prdgx" event={"ID":"c6629db6-b1af-479a-ad73-d44aba8b624f","Type":"ContainerStarted","Data":"55de50828a06263601e4d56c9dc8c62c3456d6dbc956e2204d16f6094eca0166"} Sep 08 04:01:42.860612 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:42.860594 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657cf94686-prdgx" Sep 08 04:01:42.862312 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:42.862291 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657cf94686-prdgx" Sep 08 04:01:42.873093 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:42.873055 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-pqx2g" podStartSLOduration=35.967132884 podStartE2EDuration="39.873043533s" totalImagesPullingTime="3.905910649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:01:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:42.872332252 +0000 UTC m=+76.794793028" watchObservedRunningTime="2026-09-08 04:01:42.873043533 +0000 UTC m=+76.795504271" Sep 08 04:01:42.887912 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:42.887870 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657cf94686-prdgx" podStartSLOduration=3.930763425 podStartE2EDuration="7.887857179s" totalImagesPullingTime="3.957093754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:01:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:42.88753163 +0000 UTC m=+76.809992386" watchObservedRunningTime="2026-09-08 04:01:42.887857179 +0000 UTC m=+76.810317912" Sep 08 04:01:42.901942 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:42.901907 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-6czj4" podStartSLOduration=34.98879445 podStartE2EDuration="38.901898103s" totalImagesPullingTime="3.913103653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:42.901468828 +0000 UTC m=+76.823929589" watchObservedRunningTime="2026-09-08 04:01:42.901898103 +0000 UTC m=+76.824358873" Sep 08 04:01:42.976920 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:42.976889 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b5cda57-838f-4ba0-87be-f09808d06d65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kjg2p\" (UID: \"5b5cda57-838f-4ba0-87be-f09808d06d65\") " pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:42.978922 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:42.978902 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b5cda57-838f-4ba0-87be-f09808d06d65-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kjg2p\" (UID: \"5b5cda57-838f-4ba0-87be-f09808d06d65\") " pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:43.197752 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:43.197689 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7wgqk\"" Sep 08 04:01:43.204952 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:43.204933 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kjg2p" Sep 08 04:01:43.479414 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:43.479071 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kjg2p"] Sep 08 04:01:43.481756 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:01:43.481732 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b5cda57_838f_4ba0_87be_f09808d06d65.slice/crio-29523453d73a1ab2cb01c30c04d3c48dc71c75d1b81914302133ae6bfa03b129 WatchSource:0}: Error finding container 29523453d73a1ab2cb01c30c04d3c48dc71c75d1b81914302133ae6bfa03b129: Status 404 returned error can't find the container with id 29523453d73a1ab2cb01c30c04d3c48dc71c75d1b81914302133ae6bfa03b129 Sep 08 04:01:43.866436 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:43.866397 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kjg2p" event={"ID":"5b5cda57-838f-4ba0-87be-f09808d06d65","Type":"ContainerStarted","Data":"545f4c173dec9048f23d5fe92711fb2de1040a2a69d480a1452ca45aabbbc221"} Sep 08 04:01:43.866853 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:43.866514 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kjg2p" event={"ID":"5b5cda57-838f-4ba0-87be-f09808d06d65","Type":"ContainerStarted","Data":"29523453d73a1ab2cb01c30c04d3c48dc71c75d1b81914302133ae6bfa03b129"} Sep 08 04:01:44.078566 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:44.078529 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:44.175882 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:44.175857 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:44.821596 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:44.821568 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-b8vzf" Sep 08 04:01:44.871706 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:44.871675 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kjg2p" event={"ID":"5b5cda57-838f-4ba0-87be-f09808d06d65","Type":"ContainerStarted","Data":"1ed836aa822b811c8d24ef9d4555e294be90b8be93431367ca5fe9e2a4736a70"} Sep 08 04:01:45.208304 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:45.208244 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:45.282239 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:45.282208 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:46.794619 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:46.794589 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:46.885232 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:46.885195 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:47.213476 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:47.213443 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:47.886079 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:47.886044 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kjg2p" event={"ID":"5b5cda57-838f-4ba0-87be-f09808d06d65","Type":"ContainerStarted","Data":"6f60ef947aabdd779e6a240553ccae99413d9811d41bfa7ba193592b1461b5fa"} Sep 08 04:01:47.907405 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:47.907355 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kjg2p" podStartSLOduration=33.422240023 podStartE2EDuration="36.907337151s" totalImagesPullingTime="3.485097128s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 04:01:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:47.905211842 +0000 UTC m=+81.827672607" watchObservedRunningTime="2026-09-08 04:01:47.907337151 +0000 UTC m=+81.829797888" Sep 08 04:01:48.106490 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:48.106456 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:48.180823 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:48.180758 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:48.247613 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:48.247591 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:48.349025 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:48.348996 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:48.508057 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:48.508030 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:48.702030 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:48.702002 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:48.772936 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:48.772853 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:48.893639 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:48.893531 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:49.117236 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:49.117167 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:49.687590 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:49.687541 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:50.183342 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:50.183316 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:50.283441 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:50.283417 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:51.874451 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:51.874405 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:51.974555 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:51.974525 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:52.939588 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:52.939561 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:53.236043 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.235972 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:53.308904 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.308880 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7mx8j"] Sep 08 04:01:53.311782 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.311767 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.314673 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.314646 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nrb2v\"" Sep 08 04:01:53.314762 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.314691 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 08 04:01:53.314762 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.314692 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 08 04:01:53.316098 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.316081 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 08 04:01:53.316189 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.316147 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 08 04:01:53.345461 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.345437 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b15a3b0-00f6-419e-91b5-f0cc4c166d17-node-exporter-textfile\") pod \"node-exporter-7mx8j\" (UID: \"4b15a3b0-00f6-419e-91b5-f0cc4c166d17\") " pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.345570 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.345468 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnrdv\" (UniqueName: \"kubernetes.io/projected/4b15a3b0-00f6-419e-91b5-f0cc4c166d17-kube-api-access-vnrdv\") pod \"node-exporter-7mx8j\" (UID: \"4b15a3b0-00f6-419e-91b5-f0cc4c166d17\") " pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.345570 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.345496 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b15a3b0-00f6-419e-91b5-f0cc4c166d17-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7mx8j\" (UID: \"4b15a3b0-00f6-419e-91b5-f0cc4c166d17\") " pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.345570 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.345547 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b15a3b0-00f6-419e-91b5-f0cc4c166d17-node-exporter-accelerators-collector-config\") pod \"node-exporter-7mx8j\" (UID: \"4b15a3b0-00f6-419e-91b5-f0cc4c166d17\") " pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.345723 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.345571 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b15a3b0-00f6-419e-91b5-f0cc4c166d17-node-exporter-tls\") pod \"node-exporter-7mx8j\" (UID: \"4b15a3b0-00f6-419e-91b5-f0cc4c166d17\") " pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.345723 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.345598 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b15a3b0-00f6-419e-91b5-f0cc4c166d17-sys\") pod \"node-exporter-7mx8j\" (UID: \"4b15a3b0-00f6-419e-91b5-f0cc4c166d17\") " pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.345723 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.345621 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b15a3b0-00f6-419e-91b5-f0cc4c166d17-root\") pod \"node-exporter-7mx8j\" (UID: \"4b15a3b0-00f6-419e-91b5-f0cc4c166d17\") " pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.345723 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.345642 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b15a3b0-00f6-419e-91b5-f0cc4c166d17-node-exporter-wtmp\") pod \"node-exporter-7mx8j\" (UID: \"4b15a3b0-00f6-419e-91b5-f0cc4c166d17\") " pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.345860 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.345744 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b15a3b0-00f6-419e-91b5-f0cc4c166d17-metrics-client-ca\") pod \"node-exporter-7mx8j\" (UID: \"4b15a3b0-00f6-419e-91b5-f0cc4c166d17\") " pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.389141 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.389116 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:53.446980 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.446959 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b15a3b0-00f6-419e-91b5-f0cc4c166d17-node-exporter-textfile\") pod \"node-exporter-7mx8j\" (UID: \"4b15a3b0-00f6-419e-91b5-f0cc4c166d17\") " pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.447058 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.446984 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnrdv\" (UniqueName: \"kubernetes.io/projected/4b15a3b0-00f6-419e-91b5-f0cc4c166d17-kube-api-access-vnrdv\") pod \"node-exporter-7mx8j\" (UID: \"4b15a3b0-00f6-419e-91b5-f0cc4c166d17\") " pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.447058 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.447003 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b15a3b0-00f6-419e-91b5-f0cc4c166d17-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7mx8j\" (UID: \"4b15a3b0-00f6-419e-91b5-f0cc4c166d17\") " pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.447058 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.447025 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b15a3b0-00f6-419e-91b5-f0cc4c166d17-node-exporter-accelerators-collector-config\") pod \"node-exporter-7mx8j\" (UID: \"4b15a3b0-00f6-419e-91b5-f0cc4c166d17\") " pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.447058 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.447048 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b15a3b0-00f6-419e-91b5-f0cc4c166d17-node-exporter-tls\") pod \"node-exporter-7mx8j\" (UID: \"4b15a3b0-00f6-419e-91b5-f0cc4c166d17\") " pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.447227 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.447167 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b15a3b0-00f6-419e-91b5-f0cc4c166d17-sys\") pod \"node-exporter-7mx8j\" (UID: \"4b15a3b0-00f6-419e-91b5-f0cc4c166d17\") " pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.447227 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.447215 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b15a3b0-00f6-419e-91b5-f0cc4c166d17-root\") pod \"node-exporter-7mx8j\" (UID: \"4b15a3b0-00f6-419e-91b5-f0cc4c166d17\") " pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.447324 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.447242 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b15a3b0-00f6-419e-91b5-f0cc4c166d17-node-exporter-wtmp\") pod \"node-exporter-7mx8j\" (UID: \"4b15a3b0-00f6-419e-91b5-f0cc4c166d17\") " pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.447324 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.447253 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b15a3b0-00f6-419e-91b5-f0cc4c166d17-sys\") pod \"node-exporter-7mx8j\" (UID: \"4b15a3b0-00f6-419e-91b5-f0cc4c166d17\") " pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.447324 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.447302 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b15a3b0-00f6-419e-91b5-f0cc4c166d17-node-exporter-textfile\") pod \"node-exporter-7mx8j\" (UID: \"4b15a3b0-00f6-419e-91b5-f0cc4c166d17\") " pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.447439 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.447328 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b15a3b0-00f6-419e-91b5-f0cc4c166d17-root\") pod \"node-exporter-7mx8j\" (UID: \"4b15a3b0-00f6-419e-91b5-f0cc4c166d17\") " pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.447439 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.447401 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b15a3b0-00f6-419e-91b5-f0cc4c166d17-metrics-client-ca\") pod \"node-exporter-7mx8j\" (UID: \"4b15a3b0-00f6-419e-91b5-f0cc4c166d17\") " pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.447439 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.447432 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b15a3b0-00f6-419e-91b5-f0cc4c166d17-node-exporter-wtmp\") pod \"node-exporter-7mx8j\" (UID: \"4b15a3b0-00f6-419e-91b5-f0cc4c166d17\") " pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.447612 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.447595 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b15a3b0-00f6-419e-91b5-f0cc4c166d17-node-exporter-accelerators-collector-config\") pod \"node-exporter-7mx8j\" (UID: \"4b15a3b0-00f6-419e-91b5-f0cc4c166d17\") " pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.447863 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.447845 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b15a3b0-00f6-419e-91b5-f0cc4c166d17-metrics-client-ca\") pod \"node-exporter-7mx8j\" (UID: \"4b15a3b0-00f6-419e-91b5-f0cc4c166d17\") " pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.449269 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.449251 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b15a3b0-00f6-419e-91b5-f0cc4c166d17-node-exporter-tls\") pod \"node-exporter-7mx8j\" (UID: \"4b15a3b0-00f6-419e-91b5-f0cc4c166d17\") " pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.449566 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.449550 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b15a3b0-00f6-419e-91b5-f0cc4c166d17-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7mx8j\" (UID: \"4b15a3b0-00f6-419e-91b5-f0cc4c166d17\") " pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.462954 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.462936 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnrdv\" (UniqueName: \"kubernetes.io/projected/4b15a3b0-00f6-419e-91b5-f0cc4c166d17-kube-api-access-vnrdv\") pod \"node-exporter-7mx8j\" (UID: \"4b15a3b0-00f6-419e-91b5-f0cc4c166d17\") " pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.490488 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.490438 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:53.597329 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.597303 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:53.620884 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.620861 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7mx8j" Sep 08 04:01:53.632111 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:01:53.632089 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b15a3b0_00f6_419e_91b5_f0cc4c166d17.slice/crio-a524384643eb3123938136b48fc6c7e1a58107d48be8555a24081f16d2c2a412 WatchSource:0}: Error finding container a524384643eb3123938136b48fc6c7e1a58107d48be8555a24081f16d2c2a412: Status 404 returned error can't find the container with id a524384643eb3123938136b48fc6c7e1a58107d48be8555a24081f16d2c2a412 Sep 08 04:01:53.701573 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.701548 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:53.799975 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.799936 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:53.906988 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:53.906959 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7mx8j" event={"ID":"4b15a3b0-00f6-419e-91b5-f0cc4c166d17","Type":"ContainerStarted","Data":"a524384643eb3123938136b48fc6c7e1a58107d48be8555a24081f16d2c2a412"} Sep 08 04:01:54.050040 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:54.049978 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:54.196010 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:54.195979 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:54.301886 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:54.301789 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:54.368979 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:54.368949 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:54.912026 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:54.911999 2574 generic.go:356] "Generic (PLEG): container finished" podID="4b15a3b0-00f6-419e-91b5-f0cc4c166d17" containerID="eac97e1d372be23de84dea6ed645beab447ed817de7679576bb429e0e7810aa5" exitCode=0 Sep 08 04:01:54.912130 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:54.912090 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7mx8j" event={"ID":"4b15a3b0-00f6-419e-91b5-f0cc4c166d17","Type":"ContainerDied","Data":"eac97e1d372be23de84dea6ed645beab447ed817de7679576bb429e0e7810aa5"} Sep 08 04:01:55.922226 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:55.922192 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7mx8j" event={"ID":"4b15a3b0-00f6-419e-91b5-f0cc4c166d17","Type":"ContainerStarted","Data":"e8ba15ae29880bb0e45f9b332854b3871e3153e8f93bd2dbb54e34b67fbde97d"} Sep 08 04:01:55.922672 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:55.922247 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7mx8j" event={"ID":"4b15a3b0-00f6-419e-91b5-f0cc4c166d17","Type":"ContainerStarted","Data":"53996ab9e21a32e587c3ebd9133f547e01cd5db6a53b96b425df6f96541695db"} Sep 08 04:01:55.948146 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:55.948094 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7mx8j" podStartSLOduration=1.806537059 podStartE2EDuration="2.948078891s" totalImagesPullingTime="1.082177698s" totalInitContainerRuntime="59.364134ms" isStatefulPod=true podCreationTimestamp="2026-09-08 04:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:01:55.946489112 +0000 UTC m=+89.868949858" watchObservedRunningTime="2026-09-08 04:01:55.948078891 +0000 UTC m=+89.870539629" Sep 08 04:01:56.178790 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:56.178723 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:56.277658 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:56.277634 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:57.436272 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:57.436220 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:57.930133 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:57.930102 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:58.044253 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:58.044222 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:58.467835 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:58.467792 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:59.539402 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.539365 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:59.627191 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.627158 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b57847f57-9j84g"] Sep 08 04:01:59.629626 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.629606 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b57847f57-9j84g" Sep 08 04:01:59.633462 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.633436 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 08 04:01:59.634126 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.634107 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 08 04:01:59.634208 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.634159 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-q6hs5\"" Sep 08 04:01:59.634208 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.634167 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 08 04:01:59.634312 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.634169 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 08 04:01:59.634641 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.634622 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 08 04:01:59.634709 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.634652 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 08 04:01:59.634709 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.634678 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 08 04:01:59.650035 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.650004 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b57847f57-9j84g"] Sep 08 04:01:59.691673 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.691652 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs845\" (UniqueName: \"kubernetes.io/projected/2057e6ee-35ca-45d9-bc33-831f7effe806-kube-api-access-vs845\") pod \"console-6b57847f57-9j84g\" (UID: \"2057e6ee-35ca-45d9-bc33-831f7effe806\") " pod="openshift-console/console-6b57847f57-9j84g" Sep 08 04:01:59.691769 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.691680 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2057e6ee-35ca-45d9-bc33-831f7effe806-oauth-serving-cert\") pod \"console-6b57847f57-9j84g\" (UID: \"2057e6ee-35ca-45d9-bc33-831f7effe806\") " pod="openshift-console/console-6b57847f57-9j84g" Sep 08 04:01:59.691769 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.691702 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2057e6ee-35ca-45d9-bc33-831f7effe806-console-serving-cert\") pod \"console-6b57847f57-9j84g\" (UID: \"2057e6ee-35ca-45d9-bc33-831f7effe806\") " pod="openshift-console/console-6b57847f57-9j84g" Sep 08 04:01:59.691769 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.691723 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2057e6ee-35ca-45d9-bc33-831f7effe806-console-oauth-config\") pod \"console-6b57847f57-9j84g\" (UID: \"2057e6ee-35ca-45d9-bc33-831f7effe806\") " pod="openshift-console/console-6b57847f57-9j84g" Sep 08 04:01:59.691960 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.691783 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2057e6ee-35ca-45d9-bc33-831f7effe806-console-config\") pod \"console-6b57847f57-9j84g\" (UID: \"2057e6ee-35ca-45d9-bc33-831f7effe806\") " pod="openshift-console/console-6b57847f57-9j84g" Sep 08 04:01:59.691960 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.691836 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2057e6ee-35ca-45d9-bc33-831f7effe806-service-ca\") pod \"console-6b57847f57-9j84g\" (UID: \"2057e6ee-35ca-45d9-bc33-831f7effe806\") " pod="openshift-console/console-6b57847f57-9j84g" Sep 08 04:01:59.717205 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.717180 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:59.792383 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.792325 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs845\" (UniqueName: \"kubernetes.io/projected/2057e6ee-35ca-45d9-bc33-831f7effe806-kube-api-access-vs845\") pod \"console-6b57847f57-9j84g\" (UID: \"2057e6ee-35ca-45d9-bc33-831f7effe806\") " pod="openshift-console/console-6b57847f57-9j84g" Sep 08 04:01:59.792383 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.792359 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2057e6ee-35ca-45d9-bc33-831f7effe806-oauth-serving-cert\") pod \"console-6b57847f57-9j84g\" (UID: \"2057e6ee-35ca-45d9-bc33-831f7effe806\") " pod="openshift-console/console-6b57847f57-9j84g" Sep 08 04:01:59.792383 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.792381 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2057e6ee-35ca-45d9-bc33-831f7effe806-console-serving-cert\") pod \"console-6b57847f57-9j84g\" (UID: \"2057e6ee-35ca-45d9-bc33-831f7effe806\") " pod="openshift-console/console-6b57847f57-9j84g" Sep 08 04:01:59.792632 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.792403 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2057e6ee-35ca-45d9-bc33-831f7effe806-console-oauth-config\") pod \"console-6b57847f57-9j84g\" (UID: \"2057e6ee-35ca-45d9-bc33-831f7effe806\") " pod="openshift-console/console-6b57847f57-9j84g" Sep 08 04:01:59.792632 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.792427 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2057e6ee-35ca-45d9-bc33-831f7effe806-console-config\") pod \"console-6b57847f57-9j84g\" (UID: \"2057e6ee-35ca-45d9-bc33-831f7effe806\") " pod="openshift-console/console-6b57847f57-9j84g" Sep 08 04:01:59.792632 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.792584 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2057e6ee-35ca-45d9-bc33-831f7effe806-service-ca\") pod \"console-6b57847f57-9j84g\" (UID: \"2057e6ee-35ca-45d9-bc33-831f7effe806\") " pod="openshift-console/console-6b57847f57-9j84g" Sep 08 04:01:59.793135 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.793096 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2057e6ee-35ca-45d9-bc33-831f7effe806-oauth-serving-cert\") pod \"console-6b57847f57-9j84g\" (UID: \"2057e6ee-35ca-45d9-bc33-831f7effe806\") " pod="openshift-console/console-6b57847f57-9j84g" Sep 08 04:01:59.793232 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.793191 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2057e6ee-35ca-45d9-bc33-831f7effe806-service-ca\") pod \"console-6b57847f57-9j84g\" (UID: \"2057e6ee-35ca-45d9-bc33-831f7effe806\") " pod="openshift-console/console-6b57847f57-9j84g" Sep 08 04:01:59.793328 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.793305 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2057e6ee-35ca-45d9-bc33-831f7effe806-console-config\") pod \"console-6b57847f57-9j84g\" (UID: \"2057e6ee-35ca-45d9-bc33-831f7effe806\") " pod="openshift-console/console-6b57847f57-9j84g" Sep 08 04:01:59.795004 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.794985 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2057e6ee-35ca-45d9-bc33-831f7effe806-console-oauth-config\") pod \"console-6b57847f57-9j84g\" (UID: \"2057e6ee-35ca-45d9-bc33-831f7effe806\") " pod="openshift-console/console-6b57847f57-9j84g" Sep 08 04:01:59.795258 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.795234 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2057e6ee-35ca-45d9-bc33-831f7effe806-console-serving-cert\") pod \"console-6b57847f57-9j84g\" (UID: \"2057e6ee-35ca-45d9-bc33-831f7effe806\") " pod="openshift-console/console-6b57847f57-9j84g" Sep 08 04:01:59.800599 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.800571 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs845\" (UniqueName: \"kubernetes.io/projected/2057e6ee-35ca-45d9-bc33-831f7effe806-kube-api-access-vs845\") pod \"console-6b57847f57-9j84g\" (UID: \"2057e6ee-35ca-45d9-bc33-831f7effe806\") " pod="openshift-console/console-6b57847f57-9j84g" Sep 08 04:01:59.819454 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.819422 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:01:59.848175 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.848153 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64d47bbb76-26f2f" Sep 08 04:01:59.939956 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:01:59.939926 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b57847f57-9j84g" Sep 08 04:02:00.091925 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:00.091857 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:02:00.150091 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:00.150065 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:02:00.206874 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:00.206854 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b57847f57-9j84g"] Sep 08 04:02:00.209117 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:02:00.209093 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2057e6ee_35ca_45d9_bc33_831f7effe806.slice/crio-4bdbe0c5666b4bac96586dfb5e7dbbff1b502b4aadc9494870ef2997ee434cfb WatchSource:0}: Error finding container 4bdbe0c5666b4bac96586dfb5e7dbbff1b502b4aadc9494870ef2997ee434cfb: Status 404 returned error can't find the container with id 4bdbe0c5666b4bac96586dfb5e7dbbff1b502b4aadc9494870ef2997ee434cfb Sep 08 04:02:00.940472 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:00.940442 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b57847f57-9j84g" event={"ID":"2057e6ee-35ca-45d9-bc33-831f7effe806","Type":"ContainerStarted","Data":"4bdbe0c5666b4bac96586dfb5e7dbbff1b502b4aadc9494870ef2997ee434cfb"} Sep 08 04:02:03.954323 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:03.954286 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b57847f57-9j84g" event={"ID":"2057e6ee-35ca-45d9-bc33-831f7effe806","Type":"ContainerStarted","Data":"61dadbe51cc546453b38175a5106b2a2f226b14650fe2805db63d00c04d22116"} Sep 08 04:02:03.975163 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:03.975116 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b57847f57-9j84g" podStartSLOduration=2.149706195 podStartE2EDuration="4.975100925s" totalImagesPullingTime="2.82539473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:01:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:02:03.974674654 +0000 UTC m=+97.897135411" watchObservedRunningTime="2026-09-08 04:02:03.975100925 +0000 UTC m=+97.897561663" Sep 08 04:02:04.703754 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:04.703722 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:02:04.756401 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:04.756373 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:02:08.247686 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:08.247648 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:02:08.348640 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:08.348605 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:02:09.107819 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.107785 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:02:09.195100 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.195068 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:02:09.296159 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.296124 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56864ff9b5-b7xsp"] Sep 08 04:02:09.332541 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.332518 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56864ff9b5-b7xsp"] Sep 08 04:02:09.332663 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.332619 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:02:09.340243 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.340221 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 08 04:02:09.460491 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.460441 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-console-serving-cert\") pod \"console-56864ff9b5-b7xsp\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:02:09.460491 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.460469 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-service-ca\") pod \"console-56864ff9b5-b7xsp\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:02:09.460491 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.460492 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-oauth-serving-cert\") pod \"console-56864ff9b5-b7xsp\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:02:09.460687 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.460542 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txcmc\" (UniqueName: \"kubernetes.io/projected/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-kube-api-access-txcmc\") pod \"console-56864ff9b5-b7xsp\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:02:09.460687 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.460558 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-console-oauth-config\") pod \"console-56864ff9b5-b7xsp\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:02:09.460687 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.460601 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-trusted-ca-bundle\") pod \"console-56864ff9b5-b7xsp\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:02:09.460687 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.460681 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-console-config\") pod \"console-56864ff9b5-b7xsp\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:02:09.560945 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.560919 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-console-serving-cert\") pod \"console-56864ff9b5-b7xsp\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:02:09.561043 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.560951 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-service-ca\") pod \"console-56864ff9b5-b7xsp\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:02:09.561043 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.560972 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-oauth-serving-cert\") pod \"console-56864ff9b5-b7xsp\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:02:09.561043 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.561022 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txcmc\" (UniqueName: \"kubernetes.io/projected/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-kube-api-access-txcmc\") pod \"console-56864ff9b5-b7xsp\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:02:09.561176 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.561048 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-console-oauth-config\") pod \"console-56864ff9b5-b7xsp\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:02:09.561176 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.561069 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-trusted-ca-bundle\") pod \"console-56864ff9b5-b7xsp\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:02:09.561176 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.561106 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-console-config\") pod \"console-56864ff9b5-b7xsp\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:02:09.561813 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.561789 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-oauth-serving-cert\") pod \"console-56864ff9b5-b7xsp\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:02:09.561906 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.561826 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-service-ca\") pod \"console-56864ff9b5-b7xsp\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:02:09.561906 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.561853 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-console-config\") pod \"console-56864ff9b5-b7xsp\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:02:09.562570 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.562548 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-trusted-ca-bundle\") pod \"console-56864ff9b5-b7xsp\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:02:09.563564 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.563536 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-console-serving-cert\") pod \"console-56864ff9b5-b7xsp\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:02:09.564010 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.563978 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-console-oauth-config\") pod \"console-56864ff9b5-b7xsp\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:02:09.570069 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.570046 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txcmc\" (UniqueName: \"kubernetes.io/projected/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-kube-api-access-txcmc\") pod \"console-56864ff9b5-b7xsp\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:02:09.642017 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.641991 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:02:09.845796 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.845771 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-l6l2z" Sep 08 04:02:09.940681 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.940637 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b57847f57-9j84g" Sep 08 04:02:09.940856 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.940708 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b57847f57-9j84g" Sep 08 04:02:09.941692 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.941562 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56864ff9b5-b7xsp"] Sep 08 04:02:09.944254 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:02:09.944227 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d4bea84_d1f0_47a2_b88e_a27ef3b3b78f.slice/crio-a05a2dcca9a0dedf63e4843a93ea29e59d0238ca3b022ad18945703adeba0c48 WatchSource:0}: Error finding container a05a2dcca9a0dedf63e4843a93ea29e59d0238ca3b022ad18945703adeba0c48: Status 404 returned error can't find the container with id a05a2dcca9a0dedf63e4843a93ea29e59d0238ca3b022ad18945703adeba0c48 Sep 08 04:02:09.945936 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.945912 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b57847f57-9j84g" Sep 08 04:02:09.975482 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.975434 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56864ff9b5-b7xsp" event={"ID":"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f","Type":"ContainerStarted","Data":"a05a2dcca9a0dedf63e4843a93ea29e59d0238ca3b022ad18945703adeba0c48"} Sep 08 04:02:09.983841 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:09.983818 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b57847f57-9j84g" Sep 08 04:02:10.980844 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:10.980800 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56864ff9b5-b7xsp" event={"ID":"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f","Type":"ContainerStarted","Data":"b6fb19d3b8a2d1202a4627bceaa0882f153721291269c4a7da6f987b8c5a9636"} Sep 08 04:02:11.000778 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:11.000725 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56864ff9b5-b7xsp" podStartSLOduration=2.000707891 podStartE2EDuration="2.000707891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:02:10.999449795 +0000 UTC m=+104.921910553" watchObservedRunningTime="2026-09-08 04:02:11.000707891 +0000 UTC m=+104.923168629" Sep 08 04:02:11.139473 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:11.139442 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:02:11.204586 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:11.204558 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:02:12.911828 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:12.911791 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:02:19.011866 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:19.011828 2574 generic.go:356] "Generic (PLEG): container finished" podID="56994db0-6aa0-44a7-a540-777271c022ee" containerID="714d78de5b7ae9ed6803aff29e66dda2b0dd01fcd5d09d027def1bf9894ffea5" exitCode=0 Sep 08 04:02:19.012212 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:19.011906 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xmcbr" event={"ID":"56994db0-6aa0-44a7-a540-777271c022ee","Type":"ContainerDied","Data":"714d78de5b7ae9ed6803aff29e66dda2b0dd01fcd5d09d027def1bf9894ffea5"} Sep 08 04:02:19.012272 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:19.012253 2574 scope.go:117] "RemoveContainer" containerID="714d78de5b7ae9ed6803aff29e66dda2b0dd01fcd5d09d027def1bf9894ffea5" Sep 08 04:02:19.642723 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:19.642694 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:02:19.642887 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:19.642750 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:02:19.647189 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:19.647163 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:02:20.017581 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:20.017549 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-xmcbr" event={"ID":"56994db0-6aa0-44a7-a540-777271c022ee","Type":"ContainerStarted","Data":"c7cdd7c7bf87451350560297427dfef0fa6fadd13d91c251c0c0b795893ca73c"} Sep 08 04:02:20.021109 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:20.021084 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:02:20.095911 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:20.095879 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b57847f57-9j84g"] Sep 08 04:02:21.192884 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:21.192831 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55766f96d-krtn9_e1a7f679-2737-437e-a718-53fe941b999a/router/0.log" Sep 08 04:02:21.215461 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:21.215433 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pv6qv_3e2eec1f-b2a7-43d5-a9e7-f15eafb23937/serve-healthcheck-canary/0.log" Sep 08 04:02:23.319161 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:23.319126 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:02:23.404321 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:23.404292 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:02:23.762401 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:23.762364 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:02:23.844199 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:23.844153 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:02:39.089813 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:39.089728 2574 generic.go:356] "Generic (PLEG): container finished" podID="679b83df-ec27-4bf7-902a-a53fc284973d" containerID="f290a1c1387b5df3bf04fd0b25375a9dc8f9a377a0a57c1a2c3c70d67a865bee" exitCode=0 Sep 08 04:02:39.090181 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:39.089817 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p59lp" event={"ID":"679b83df-ec27-4bf7-902a-a53fc284973d","Type":"ContainerDied","Data":"f290a1c1387b5df3bf04fd0b25375a9dc8f9a377a0a57c1a2c3c70d67a865bee"} Sep 08 04:02:39.090181 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:39.090130 2574 scope.go:117] "RemoveContainer" containerID="f290a1c1387b5df3bf04fd0b25375a9dc8f9a377a0a57c1a2c3c70d67a865bee" Sep 08 04:02:40.094764 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:40.094732 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-p59lp" event={"ID":"679b83df-ec27-4bf7-902a-a53fc284973d","Type":"ContainerStarted","Data":"0b76a127f6d6b9daa781f82b3d3c9602753f6148f04f115ed4078534f015ab77"} Sep 08 04:02:45.117400 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:45.117340 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b57847f57-9j84g" podUID="2057e6ee-35ca-45d9-bc33-831f7effe806" containerName="console" containerID="cri-o://61dadbe51cc546453b38175a5106b2a2f226b14650fe2805db63d00c04d22116" gracePeriod=15 Sep 08 04:02:45.224008 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:45.223977 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:02:45.324361 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:45.324337 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:02:45.351712 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:45.351691 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b57847f57-9j84g" Sep 08 04:02:45.426428 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:45.426373 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:02:45.524666 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:45.524641 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vs845\" (UniqueName: \"kubernetes.io/projected/2057e6ee-35ca-45d9-bc33-831f7effe806-kube-api-access-vs845\") pod \"2057e6ee-35ca-45d9-bc33-831f7effe806\" (UID: \"2057e6ee-35ca-45d9-bc33-831f7effe806\") " Sep 08 04:02:45.524760 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:45.524681 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2057e6ee-35ca-45d9-bc33-831f7effe806-console-config\") pod \"2057e6ee-35ca-45d9-bc33-831f7effe806\" (UID: \"2057e6ee-35ca-45d9-bc33-831f7effe806\") " Sep 08 04:02:45.524760 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:45.524720 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2057e6ee-35ca-45d9-bc33-831f7effe806-console-oauth-config\") pod \"2057e6ee-35ca-45d9-bc33-831f7effe806\" (UID: \"2057e6ee-35ca-45d9-bc33-831f7effe806\") " Sep 08 04:02:45.524854 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:45.524825 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2057e6ee-35ca-45d9-bc33-831f7effe806-console-serving-cert\") pod \"2057e6ee-35ca-45d9-bc33-831f7effe806\" (UID: \"2057e6ee-35ca-45d9-bc33-831f7effe806\") " Sep 08 04:02:45.524900 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:45.524878 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2057e6ee-35ca-45d9-bc33-831f7effe806-oauth-serving-cert\") pod \"2057e6ee-35ca-45d9-bc33-831f7effe806\" (UID: \"2057e6ee-35ca-45d9-bc33-831f7effe806\") " Sep 08 04:02:45.524949 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:45.524911 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2057e6ee-35ca-45d9-bc33-831f7effe806-service-ca\") pod \"2057e6ee-35ca-45d9-bc33-831f7effe806\" (UID: \"2057e6ee-35ca-45d9-bc33-831f7effe806\") " Sep 08 04:02:45.525226 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:45.525129 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2057e6ee-35ca-45d9-bc33-831f7effe806-console-config" (OuterVolumeSpecName: "console-config") pod "2057e6ee-35ca-45d9-bc33-831f7effe806" (UID: "2057e6ee-35ca-45d9-bc33-831f7effe806"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:02:45.525403 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:45.525312 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2057e6ee-35ca-45d9-bc33-831f7effe806-service-ca" (OuterVolumeSpecName: "service-ca") pod "2057e6ee-35ca-45d9-bc33-831f7effe806" (UID: "2057e6ee-35ca-45d9-bc33-831f7effe806"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:02:45.525403 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:45.525309 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2057e6ee-35ca-45d9-bc33-831f7effe806-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2057e6ee-35ca-45d9-bc33-831f7effe806" (UID: "2057e6ee-35ca-45d9-bc33-831f7effe806"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:02:45.526976 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:45.526949 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2057e6ee-35ca-45d9-bc33-831f7effe806-kube-api-access-vs845" (OuterVolumeSpecName: "kube-api-access-vs845") pod "2057e6ee-35ca-45d9-bc33-831f7effe806" (UID: "2057e6ee-35ca-45d9-bc33-831f7effe806"). InnerVolumeSpecName "kube-api-access-vs845". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:02:45.527115 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:45.527096 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2057e6ee-35ca-45d9-bc33-831f7effe806-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2057e6ee-35ca-45d9-bc33-831f7effe806" (UID: "2057e6ee-35ca-45d9-bc33-831f7effe806"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 04:02:45.527163 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:45.527134 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2057e6ee-35ca-45d9-bc33-831f7effe806-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2057e6ee-35ca-45d9-bc33-831f7effe806" (UID: "2057e6ee-35ca-45d9-bc33-831f7effe806"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 04:02:45.625846 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:45.625823 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2057e6ee-35ca-45d9-bc33-831f7effe806-console-oauth-config\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:02:45.625846 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:45.625842 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2057e6ee-35ca-45d9-bc33-831f7effe806-console-serving-cert\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:02:45.625955 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:45.625852 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2057e6ee-35ca-45d9-bc33-831f7effe806-oauth-serving-cert\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:02:45.625955 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:45.625862 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2057e6ee-35ca-45d9-bc33-831f7effe806-service-ca\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:02:45.625955 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:45.625871 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs845\" (UniqueName: \"kubernetes.io/projected/2057e6ee-35ca-45d9-bc33-831f7effe806-kube-api-access-vs845\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:02:45.625955 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:45.625879 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2057e6ee-35ca-45d9-bc33-831f7effe806-console-config\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:02:46.117674 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:46.117644 2574 generic.go:356] "Generic (PLEG): container finished" podID="2057e6ee-35ca-45d9-bc33-831f7effe806" containerID="61dadbe51cc546453b38175a5106b2a2f226b14650fe2805db63d00c04d22116" exitCode=0 Sep 08 04:02:46.118008 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:46.117688 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b57847f57-9j84g" event={"ID":"2057e6ee-35ca-45d9-bc33-831f7effe806","Type":"ContainerDied","Data":"61dadbe51cc546453b38175a5106b2a2f226b14650fe2805db63d00c04d22116"} Sep 08 04:02:46.118008 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:46.117694 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b57847f57-9j84g" Sep 08 04:02:46.118008 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:46.117729 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b57847f57-9j84g" event={"ID":"2057e6ee-35ca-45d9-bc33-831f7effe806","Type":"ContainerDied","Data":"4bdbe0c5666b4bac96586dfb5e7dbbff1b502b4aadc9494870ef2997ee434cfb"} Sep 08 04:02:46.118008 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:46.117750 2574 scope.go:117] "RemoveContainer" containerID="61dadbe51cc546453b38175a5106b2a2f226b14650fe2805db63d00c04d22116" Sep 08 04:02:46.126573 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:46.126553 2574 scope.go:117] "RemoveContainer" containerID="61dadbe51cc546453b38175a5106b2a2f226b14650fe2805db63d00c04d22116" Sep 08 04:02:46.126848 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:02:46.126827 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61dadbe51cc546453b38175a5106b2a2f226b14650fe2805db63d00c04d22116\": container with ID starting with 61dadbe51cc546453b38175a5106b2a2f226b14650fe2805db63d00c04d22116 not found: ID does not exist" containerID="61dadbe51cc546453b38175a5106b2a2f226b14650fe2805db63d00c04d22116" Sep 08 04:02:46.126894 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:46.126856 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61dadbe51cc546453b38175a5106b2a2f226b14650fe2805db63d00c04d22116"} err="failed to get container status \"61dadbe51cc546453b38175a5106b2a2f226b14650fe2805db63d00c04d22116\": rpc error: code = NotFound desc = could not find container \"61dadbe51cc546453b38175a5106b2a2f226b14650fe2805db63d00c04d22116\": container with ID starting with 61dadbe51cc546453b38175a5106b2a2f226b14650fe2805db63d00c04d22116 not found: ID does not exist" Sep 08 04:02:46.138211 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:46.138187 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b57847f57-9j84g"] Sep 08 04:02:46.141662 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:46.141641 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b57847f57-9j84g"] Sep 08 04:02:46.566156 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:02:46.566131 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2057e6ee-35ca-45d9-bc33-831f7effe806" path="/var/lib/kubelet/pods/2057e6ee-35ca-45d9-bc33-831f7effe806/volumes" Sep 08 04:03:12.618852 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:12.618817 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:03:12.914821 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:12.914746 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:03:20.670176 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.670101 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65cbbcc8d8-vj8s5"] Sep 08 04:03:20.670695 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.670494 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2057e6ee-35ca-45d9-bc33-831f7effe806" containerName="console" Sep 08 04:03:20.670695 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.670531 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2057e6ee-35ca-45d9-bc33-831f7effe806" containerName="console" Sep 08 04:03:20.670695 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.670630 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2057e6ee-35ca-45d9-bc33-831f7effe806" containerName="console" Sep 08 04:03:20.673585 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.673560 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:03:20.684639 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.684618 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65cbbcc8d8-vj8s5"] Sep 08 04:03:20.743305 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.743278 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:03:20.767074 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.767055 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c112134-5975-45c6-997d-6823e45a6d6a-trusted-ca-bundle\") pod \"console-65cbbcc8d8-vj8s5\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:03:20.767143 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.767082 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5k6g\" (UniqueName: \"kubernetes.io/projected/6c112134-5975-45c6-997d-6823e45a6d6a-kube-api-access-f5k6g\") pod \"console-65cbbcc8d8-vj8s5\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:03:20.767143 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.767112 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6c112134-5975-45c6-997d-6823e45a6d6a-console-config\") pod \"console-65cbbcc8d8-vj8s5\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:03:20.767213 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.767177 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c112134-5975-45c6-997d-6823e45a6d6a-oauth-serving-cert\") pod \"console-65cbbcc8d8-vj8s5\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:03:20.767245 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.767233 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6c112134-5975-45c6-997d-6823e45a6d6a-service-ca\") pod \"console-65cbbcc8d8-vj8s5\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:03:20.767276 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.767258 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c112134-5975-45c6-997d-6823e45a6d6a-console-oauth-config\") pod \"console-65cbbcc8d8-vj8s5\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:03:20.767306 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.767280 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c112134-5975-45c6-997d-6823e45a6d6a-console-serving-cert\") pod \"console-65cbbcc8d8-vj8s5\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:03:20.842646 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.842620 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:03:20.867748 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.867728 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c112134-5975-45c6-997d-6823e45a6d6a-console-serving-cert\") pod \"console-65cbbcc8d8-vj8s5\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:03:20.867821 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.867760 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c112134-5975-45c6-997d-6823e45a6d6a-trusted-ca-bundle\") pod \"console-65cbbcc8d8-vj8s5\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:03:20.867821 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.867780 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5k6g\" (UniqueName: \"kubernetes.io/projected/6c112134-5975-45c6-997d-6823e45a6d6a-kube-api-access-f5k6g\") pod \"console-65cbbcc8d8-vj8s5\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:03:20.867821 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.867807 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6c112134-5975-45c6-997d-6823e45a6d6a-console-config\") pod \"console-65cbbcc8d8-vj8s5\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:03:20.867967 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.867826 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c112134-5975-45c6-997d-6823e45a6d6a-oauth-serving-cert\") pod \"console-65cbbcc8d8-vj8s5\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:03:20.867967 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.867860 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6c112134-5975-45c6-997d-6823e45a6d6a-service-ca\") pod \"console-65cbbcc8d8-vj8s5\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:03:20.867967 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.867876 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c112134-5975-45c6-997d-6823e45a6d6a-console-oauth-config\") pod \"console-65cbbcc8d8-vj8s5\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:03:20.868560 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.868539 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c112134-5975-45c6-997d-6823e45a6d6a-oauth-serving-cert\") pod \"console-65cbbcc8d8-vj8s5\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:03:20.868654 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.868598 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6c112134-5975-45c6-997d-6823e45a6d6a-service-ca\") pod \"console-65cbbcc8d8-vj8s5\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:03:20.868654 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.868573 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6c112134-5975-45c6-997d-6823e45a6d6a-console-config\") pod \"console-65cbbcc8d8-vj8s5\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:03:20.868733 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.868664 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c112134-5975-45c6-997d-6823e45a6d6a-trusted-ca-bundle\") pod \"console-65cbbcc8d8-vj8s5\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:03:20.870215 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.870197 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c112134-5975-45c6-997d-6823e45a6d6a-console-oauth-config\") pod \"console-65cbbcc8d8-vj8s5\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:03:20.870326 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.870310 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c112134-5975-45c6-997d-6823e45a6d6a-console-serving-cert\") pod \"console-65cbbcc8d8-vj8s5\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:03:20.877742 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.877723 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5k6g\" (UniqueName: \"kubernetes.io/projected/6c112134-5975-45c6-997d-6823e45a6d6a-kube-api-access-f5k6g\") pod \"console-65cbbcc8d8-vj8s5\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:03:20.981962 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:20.981908 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:03:21.297720 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:21.297699 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65cbbcc8d8-vj8s5"] Sep 08 04:03:21.299847 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:03:21.299825 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c112134_5975_45c6_997d_6823e45a6d6a.slice/crio-dd7c0dba342620a755ea895917f407bd7f435cc574be6fce883206d726599d30 WatchSource:0}: Error finding container dd7c0dba342620a755ea895917f407bd7f435cc574be6fce883206d726599d30: Status 404 returned error can't find the container with id dd7c0dba342620a755ea895917f407bd7f435cc574be6fce883206d726599d30 Sep 08 04:03:22.241937 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:22.241906 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65cbbcc8d8-vj8s5" event={"ID":"6c112134-5975-45c6-997d-6823e45a6d6a","Type":"ContainerStarted","Data":"2c04dafd1278378c4713e1d35d52902de0b13c0460ae444e67323233ac706fa1"} Sep 08 04:03:22.242253 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:22.241951 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65cbbcc8d8-vj8s5" event={"ID":"6c112134-5975-45c6-997d-6823e45a6d6a","Type":"ContainerStarted","Data":"dd7c0dba342620a755ea895917f407bd7f435cc574be6fce883206d726599d30"} Sep 08 04:03:22.262956 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:22.262913 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65cbbcc8d8-vj8s5" podStartSLOduration=2.26289798 podStartE2EDuration="2.26289798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:03:22.262192712 +0000 UTC m=+176.184653464" watchObservedRunningTime="2026-09-08 04:03:22.26289798 +0000 UTC m=+176.185358719" Sep 08 04:03:30.982058 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:30.982024 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:03:30.982420 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:30.982071 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:03:30.986373 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:30.986350 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:03:31.277186 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:31.277156 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:03:31.318068 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:31.318038 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56864ff9b5-b7xsp"] Sep 08 04:03:31.468629 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:31.468598 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:03:31.570837 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:31.570756 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:03:51.332799 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:51.332766 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:03:51.393108 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:51.393073 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:03:56.338531 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:56.338445 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56864ff9b5-b7xsp" podUID="9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f" containerName="console" containerID="cri-o://b6fb19d3b8a2d1202a4627bceaa0882f153721291269c4a7da6f987b8c5a9636" gracePeriod=15 Sep 08 04:03:56.575193 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:56.575164 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:03:56.717828 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:56.717731 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-oauth-serving-cert\") pod \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " Sep 08 04:03:56.718003 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:56.717863 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txcmc\" (UniqueName: \"kubernetes.io/projected/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-kube-api-access-txcmc\") pod \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " Sep 08 04:03:56.718003 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:56.717970 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-console-serving-cert\") pod \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " Sep 08 04:03:56.718120 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:56.718013 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-console-oauth-config\") pod \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " Sep 08 04:03:56.718120 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:56.718046 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-service-ca\") pod \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " Sep 08 04:03:56.718120 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:56.718054 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f" (UID: "9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:03:56.718120 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:56.718090 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-console-config\") pod \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " Sep 08 04:03:56.718120 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:56.718117 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-trusted-ca-bundle\") pod \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\" (UID: \"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f\") " Sep 08 04:03:56.718346 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:56.718327 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-oauth-serving-cert\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:03:56.718534 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:56.718475 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-service-ca" (OuterVolumeSpecName: "service-ca") pod "9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f" (UID: "9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:03:56.718638 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:56.718558 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-console-config" (OuterVolumeSpecName: "console-config") pod "9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f" (UID: "9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:03:56.718695 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:56.718647 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f" (UID: "9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:03:56.720059 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:56.720032 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-kube-api-access-txcmc" (OuterVolumeSpecName: "kube-api-access-txcmc") pod "9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f" (UID: "9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f"). InnerVolumeSpecName "kube-api-access-txcmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:03:56.720059 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:56.720043 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f" (UID: "9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 04:03:56.720192 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:56.720159 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f" (UID: "9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 04:03:56.819403 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:56.819359 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-service-ca\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:03:56.819403 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:56.819393 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-console-config\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:03:56.819403 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:56.819403 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-trusted-ca-bundle\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:03:56.819403 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:56.819412 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txcmc\" (UniqueName: \"kubernetes.io/projected/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-kube-api-access-txcmc\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:03:56.819687 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:56.819422 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-console-serving-cert\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:03:56.819687 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:56.819430 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f-console-oauth-config\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:03:57.365881 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:57.365845 2574 generic.go:356] "Generic (PLEG): container finished" podID="9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f" containerID="b6fb19d3b8a2d1202a4627bceaa0882f153721291269c4a7da6f987b8c5a9636" exitCode=0 Sep 08 04:03:57.366281 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:57.365892 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56864ff9b5-b7xsp" Sep 08 04:03:57.366281 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:57.365904 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56864ff9b5-b7xsp" event={"ID":"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f","Type":"ContainerDied","Data":"b6fb19d3b8a2d1202a4627bceaa0882f153721291269c4a7da6f987b8c5a9636"} Sep 08 04:03:57.366281 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:57.365946 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56864ff9b5-b7xsp" event={"ID":"9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f","Type":"ContainerDied","Data":"a05a2dcca9a0dedf63e4843a93ea29e59d0238ca3b022ad18945703adeba0c48"} Sep 08 04:03:57.366281 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:57.365965 2574 scope.go:117] "RemoveContainer" containerID="b6fb19d3b8a2d1202a4627bceaa0882f153721291269c4a7da6f987b8c5a9636" Sep 08 04:03:57.374248 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:57.374230 2574 scope.go:117] "RemoveContainer" containerID="b6fb19d3b8a2d1202a4627bceaa0882f153721291269c4a7da6f987b8c5a9636" Sep 08 04:03:57.374533 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:03:57.374497 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6fb19d3b8a2d1202a4627bceaa0882f153721291269c4a7da6f987b8c5a9636\": container with ID starting with b6fb19d3b8a2d1202a4627bceaa0882f153721291269c4a7da6f987b8c5a9636 not found: ID does not exist" containerID="b6fb19d3b8a2d1202a4627bceaa0882f153721291269c4a7da6f987b8c5a9636" Sep 08 04:03:57.374610 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:57.374546 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6fb19d3b8a2d1202a4627bceaa0882f153721291269c4a7da6f987b8c5a9636"} err="failed to get container status \"b6fb19d3b8a2d1202a4627bceaa0882f153721291269c4a7da6f987b8c5a9636\": rpc error: code = NotFound desc = could not find container \"b6fb19d3b8a2d1202a4627bceaa0882f153721291269c4a7da6f987b8c5a9636\": container with ID starting with b6fb19d3b8a2d1202a4627bceaa0882f153721291269c4a7da6f987b8c5a9636 not found: ID does not exist" Sep 08 04:03:57.393478 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:57.393452 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56864ff9b5-b7xsp"] Sep 08 04:03:57.402058 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:57.402032 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56864ff9b5-b7xsp"] Sep 08 04:03:58.566331 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:03:58.566294 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f" path="/var/lib/kubelet/pods/9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f/volumes" Sep 08 04:04:27.802017 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:27.801984 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:04:27.866943 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:27.866911 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:04:28.155840 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:28.155787 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:04:31.940069 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:31.940036 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75dd5cddb5-n9dc7"] Sep 08 04:04:31.940410 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:31.940283 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f" containerName="console" Sep 08 04:04:31.940410 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:31.940295 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f" containerName="console" Sep 08 04:04:31.940410 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:31.940350 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d4bea84-d1f0-47a2-b88e-a27ef3b3b78f" containerName="console" Sep 08 04:04:31.944116 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:31.944099 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:04:31.953623 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:31.953594 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75dd5cddb5-n9dc7"] Sep 08 04:04:31.963619 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:31.963597 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d42e70e1-3982-4d6d-b254-564e116de345-console-oauth-config\") pod \"console-75dd5cddb5-n9dc7\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:04:31.963731 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:31.963623 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz8l7\" (UniqueName: \"kubernetes.io/projected/d42e70e1-3982-4d6d-b254-564e116de345-kube-api-access-lz8l7\") pod \"console-75dd5cddb5-n9dc7\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:04:31.963731 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:31.963653 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d42e70e1-3982-4d6d-b254-564e116de345-console-serving-cert\") pod \"console-75dd5cddb5-n9dc7\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:04:31.963731 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:31.963692 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d42e70e1-3982-4d6d-b254-564e116de345-trusted-ca-bundle\") pod \"console-75dd5cddb5-n9dc7\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:04:31.963838 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:31.963731 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d42e70e1-3982-4d6d-b254-564e116de345-oauth-serving-cert\") pod \"console-75dd5cddb5-n9dc7\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:04:31.963838 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:31.963768 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d42e70e1-3982-4d6d-b254-564e116de345-console-config\") pod \"console-75dd5cddb5-n9dc7\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:04:31.963838 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:31.963797 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d42e70e1-3982-4d6d-b254-564e116de345-service-ca\") pod \"console-75dd5cddb5-n9dc7\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:04:32.013643 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:32.013618 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:04:32.064751 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:32.064721 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d42e70e1-3982-4d6d-b254-564e116de345-oauth-serving-cert\") pod \"console-75dd5cddb5-n9dc7\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:04:32.064852 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:32.064757 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d42e70e1-3982-4d6d-b254-564e116de345-console-config\") pod \"console-75dd5cddb5-n9dc7\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:04:32.064852 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:32.064809 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d42e70e1-3982-4d6d-b254-564e116de345-service-ca\") pod \"console-75dd5cddb5-n9dc7\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:04:32.064938 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:32.064849 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d42e70e1-3982-4d6d-b254-564e116de345-console-oauth-config\") pod \"console-75dd5cddb5-n9dc7\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:04:32.064938 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:32.064874 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz8l7\" (UniqueName: \"kubernetes.io/projected/d42e70e1-3982-4d6d-b254-564e116de345-kube-api-access-lz8l7\") pod \"console-75dd5cddb5-n9dc7\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:04:32.064938 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:32.064909 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d42e70e1-3982-4d6d-b254-564e116de345-console-serving-cert\") pod \"console-75dd5cddb5-n9dc7\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:04:32.064938 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:32.064932 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d42e70e1-3982-4d6d-b254-564e116de345-trusted-ca-bundle\") pod \"console-75dd5cddb5-n9dc7\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:04:32.065686 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:32.065655 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d42e70e1-3982-4d6d-b254-564e116de345-oauth-serving-cert\") pod \"console-75dd5cddb5-n9dc7\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:04:32.065686 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:32.065671 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d42e70e1-3982-4d6d-b254-564e116de345-service-ca\") pod \"console-75dd5cddb5-n9dc7\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:04:32.065845 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:32.065730 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d42e70e1-3982-4d6d-b254-564e116de345-console-config\") pod \"console-75dd5cddb5-n9dc7\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:04:32.065921 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:32.065903 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d42e70e1-3982-4d6d-b254-564e116de345-trusted-ca-bundle\") pod \"console-75dd5cddb5-n9dc7\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:04:32.067942 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:32.067919 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d42e70e1-3982-4d6d-b254-564e116de345-console-oauth-config\") pod \"console-75dd5cddb5-n9dc7\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:04:32.067942 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:32.067933 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d42e70e1-3982-4d6d-b254-564e116de345-console-serving-cert\") pod \"console-75dd5cddb5-n9dc7\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:04:32.069274 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:32.069251 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:04:32.072343 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:32.072323 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz8l7\" (UniqueName: \"kubernetes.io/projected/d42e70e1-3982-4d6d-b254-564e116de345-kube-api-access-lz8l7\") pod \"console-75dd5cddb5-n9dc7\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:04:32.141324 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:32.141288 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:04:32.197052 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:32.196993 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:04:32.254551 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:32.254528 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:04:32.527178 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:32.527094 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75dd5cddb5-n9dc7"] Sep 08 04:04:32.529804 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:04:32.529779 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd42e70e1_3982_4d6d_b254_564e116de345.slice/crio-93734bd53bbb526dccc7a907734ac5f438d82d39b49111b11e77710a21ea4a53 WatchSource:0}: Error finding container 93734bd53bbb526dccc7a907734ac5f438d82d39b49111b11e77710a21ea4a53: Status 404 returned error can't find the container with id 93734bd53bbb526dccc7a907734ac5f438d82d39b49111b11e77710a21ea4a53 Sep 08 04:04:32.755460 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:32.755419 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:04:32.843684 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:32.843611 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:04:33.497733 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:33.497709 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75dd5cddb5-n9dc7" event={"ID":"d42e70e1-3982-4d6d-b254-564e116de345","Type":"ContainerStarted","Data":"825612cb1f2a0c38292c9ec052d9e091e0dc894e03c4194a305bf00948a1b995"} Sep 08 04:04:33.498138 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:33.497757 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75dd5cddb5-n9dc7" event={"ID":"d42e70e1-3982-4d6d-b254-564e116de345","Type":"ContainerStarted","Data":"93734bd53bbb526dccc7a907734ac5f438d82d39b49111b11e77710a21ea4a53"} Sep 08 04:04:33.518327 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:33.518282 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75dd5cddb5-n9dc7" podStartSLOduration=2.5182684269999998 podStartE2EDuration="2.518268427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:04:33.517798008 +0000 UTC m=+247.440258759" watchObservedRunningTime="2026-09-08 04:04:33.518268427 +0000 UTC m=+247.440729166" Sep 08 04:04:39.844256 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:39.844172 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:04:39.945237 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:39.945201 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:04:42.254623 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:42.254584 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:04:42.255043 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:42.254646 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:04:42.259239 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:42.259218 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:04:42.530550 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:42.530476 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:04:42.578857 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:42.578831 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65cbbcc8d8-vj8s5"] Sep 08 04:04:42.700676 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:42.700644 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:04:42.801520 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:42.801427 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:04:42.854947 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:42.854909 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:04:53.250232 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:53.250162 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:04:53.352298 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:04:53.352274 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:05:07.605700 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:07.605648 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65cbbcc8d8-vj8s5" podUID="6c112134-5975-45c6-997d-6823e45a6d6a" containerName="console" containerID="cri-o://2c04dafd1278378c4713e1d35d52902de0b13c0460ae444e67323233ac706fa1" gracePeriod=15 Sep 08 04:05:07.843348 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:07.843326 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:05:07.911822 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:07.911767 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c112134-5975-45c6-997d-6823e45a6d6a-trusted-ca-bundle\") pod \"6c112134-5975-45c6-997d-6823e45a6d6a\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " Sep 08 04:05:07.911822 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:07.911816 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6c112134-5975-45c6-997d-6823e45a6d6a-console-config\") pod \"6c112134-5975-45c6-997d-6823e45a6d6a\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " Sep 08 04:05:07.911988 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:07.911839 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c112134-5975-45c6-997d-6823e45a6d6a-oauth-serving-cert\") pod \"6c112134-5975-45c6-997d-6823e45a6d6a\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " Sep 08 04:05:07.911988 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:07.911972 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5k6g\" (UniqueName: \"kubernetes.io/projected/6c112134-5975-45c6-997d-6823e45a6d6a-kube-api-access-f5k6g\") pod \"6c112134-5975-45c6-997d-6823e45a6d6a\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " Sep 08 04:05:07.912103 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:07.912022 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c112134-5975-45c6-997d-6823e45a6d6a-console-serving-cert\") pod \"6c112134-5975-45c6-997d-6823e45a6d6a\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " Sep 08 04:05:07.912103 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:07.912050 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6c112134-5975-45c6-997d-6823e45a6d6a-service-ca\") pod \"6c112134-5975-45c6-997d-6823e45a6d6a\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " Sep 08 04:05:07.912103 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:07.912079 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c112134-5975-45c6-997d-6823e45a6d6a-console-oauth-config\") pod \"6c112134-5975-45c6-997d-6823e45a6d6a\" (UID: \"6c112134-5975-45c6-997d-6823e45a6d6a\") " Sep 08 04:05:07.912243 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:07.912149 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c112134-5975-45c6-997d-6823e45a6d6a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6c112134-5975-45c6-997d-6823e45a6d6a" (UID: "6c112134-5975-45c6-997d-6823e45a6d6a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:05:07.912243 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:07.912187 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c112134-5975-45c6-997d-6823e45a6d6a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6c112134-5975-45c6-997d-6823e45a6d6a" (UID: "6c112134-5975-45c6-997d-6823e45a6d6a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:05:07.912243 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:07.912196 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c112134-5975-45c6-997d-6823e45a6d6a-console-config" (OuterVolumeSpecName: "console-config") pod "6c112134-5975-45c6-997d-6823e45a6d6a" (UID: "6c112134-5975-45c6-997d-6823e45a6d6a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:05:07.912475 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:07.912446 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c112134-5975-45c6-997d-6823e45a6d6a-service-ca" (OuterVolumeSpecName: "service-ca") pod "6c112134-5975-45c6-997d-6823e45a6d6a" (UID: "6c112134-5975-45c6-997d-6823e45a6d6a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:05:07.912475 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:07.912467 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6c112134-5975-45c6-997d-6823e45a6d6a-console-config\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:05:07.912591 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:07.912487 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6c112134-5975-45c6-997d-6823e45a6d6a-oauth-serving-cert\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:05:07.912591 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:07.912528 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c112134-5975-45c6-997d-6823e45a6d6a-trusted-ca-bundle\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:05:07.913999 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:07.913976 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c112134-5975-45c6-997d-6823e45a6d6a-kube-api-access-f5k6g" (OuterVolumeSpecName: "kube-api-access-f5k6g") pod "6c112134-5975-45c6-997d-6823e45a6d6a" (UID: "6c112134-5975-45c6-997d-6823e45a6d6a"). InnerVolumeSpecName "kube-api-access-f5k6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:05:07.914067 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:07.913981 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c112134-5975-45c6-997d-6823e45a6d6a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6c112134-5975-45c6-997d-6823e45a6d6a" (UID: "6c112134-5975-45c6-997d-6823e45a6d6a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 04:05:07.914106 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:07.914078 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c112134-5975-45c6-997d-6823e45a6d6a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6c112134-5975-45c6-997d-6823e45a6d6a" (UID: "6c112134-5975-45c6-997d-6823e45a6d6a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 04:05:08.013270 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:08.013245 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5k6g\" (UniqueName: \"kubernetes.io/projected/6c112134-5975-45c6-997d-6823e45a6d6a-kube-api-access-f5k6g\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:05:08.013270 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:08.013268 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c112134-5975-45c6-997d-6823e45a6d6a-console-serving-cert\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:05:08.013390 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:08.013281 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6c112134-5975-45c6-997d-6823e45a6d6a-service-ca\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:05:08.013390 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:08.013293 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6c112134-5975-45c6-997d-6823e45a6d6a-console-oauth-config\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:05:08.610119 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:08.610094 2574 generic.go:356] "Generic (PLEG): container finished" podID="6c112134-5975-45c6-997d-6823e45a6d6a" containerID="2c04dafd1278378c4713e1d35d52902de0b13c0460ae444e67323233ac706fa1" exitCode=0 Sep 08 04:05:08.610386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:08.610125 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65cbbcc8d8-vj8s5" event={"ID":"6c112134-5975-45c6-997d-6823e45a6d6a","Type":"ContainerDied","Data":"2c04dafd1278378c4713e1d35d52902de0b13c0460ae444e67323233ac706fa1"} Sep 08 04:05:08.610386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:08.610137 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65cbbcc8d8-vj8s5" Sep 08 04:05:08.610386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:08.610155 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65cbbcc8d8-vj8s5" event={"ID":"6c112134-5975-45c6-997d-6823e45a6d6a","Type":"ContainerDied","Data":"dd7c0dba342620a755ea895917f407bd7f435cc574be6fce883206d726599d30"} Sep 08 04:05:08.610386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:08.610177 2574 scope.go:117] "RemoveContainer" containerID="2c04dafd1278378c4713e1d35d52902de0b13c0460ae444e67323233ac706fa1" Sep 08 04:05:08.617708 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:08.617694 2574 scope.go:117] "RemoveContainer" containerID="2c04dafd1278378c4713e1d35d52902de0b13c0460ae444e67323233ac706fa1" Sep 08 04:05:08.617993 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:05:08.617975 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c04dafd1278378c4713e1d35d52902de0b13c0460ae444e67323233ac706fa1\": container with ID starting with 2c04dafd1278378c4713e1d35d52902de0b13c0460ae444e67323233ac706fa1 not found: ID does not exist" containerID="2c04dafd1278378c4713e1d35d52902de0b13c0460ae444e67323233ac706fa1" Sep 08 04:05:08.618574 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:08.617999 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c04dafd1278378c4713e1d35d52902de0b13c0460ae444e67323233ac706fa1"} err="failed to get container status \"2c04dafd1278378c4713e1d35d52902de0b13c0460ae444e67323233ac706fa1\": rpc error: code = NotFound desc = could not find container \"2c04dafd1278378c4713e1d35d52902de0b13c0460ae444e67323233ac706fa1\": container with ID starting with 2c04dafd1278378c4713e1d35d52902de0b13c0460ae444e67323233ac706fa1 not found: ID does not exist" Sep 08 04:05:08.633211 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:08.633188 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65cbbcc8d8-vj8s5"] Sep 08 04:05:08.636776 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:08.636757 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65cbbcc8d8-vj8s5"] Sep 08 04:05:10.566606 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:10.566565 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c112134-5975-45c6-997d-6823e45a6d6a" path="/var/lib/kubelet/pods/6c112134-5975-45c6-997d-6823e45a6d6a/volumes" Sep 08 04:05:26.440865 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:26.440837 2574 kubelet.go:1648] "Image garbage collection succeeded" Sep 08 04:05:35.792661 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:35.792626 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:05:36.544543 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:36.544500 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9"] Sep 08 04:05:36.544836 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:36.544817 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c112134-5975-45c6-997d-6823e45a6d6a" containerName="console" Sep 08 04:05:36.544836 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:36.544837 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c112134-5975-45c6-997d-6823e45a6d6a" containerName="console" Sep 08 04:05:36.545016 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:36.544898 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c112134-5975-45c6-997d-6823e45a6d6a" containerName="console" Sep 08 04:05:36.548129 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:36.548110 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9" Sep 08 04:05:36.550836 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:36.550820 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 04:05:36.550951 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:36.550857 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-d97f6\"" Sep 08 04:05:36.551807 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:36.551793 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 04:05:36.556984 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:36.556965 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9"] Sep 08 04:05:36.594519 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:36.594484 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6nbj\" (UniqueName: \"kubernetes.io/projected/2f883400-b336-4ccf-88c0-7fff2a89cf06-kube-api-access-x6nbj\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9\" (UID: \"2f883400-b336-4ccf-88c0-7fff2a89cf06\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9" Sep 08 04:05:36.594616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:36.594528 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f883400-b336-4ccf-88c0-7fff2a89cf06-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9\" (UID: \"2f883400-b336-4ccf-88c0-7fff2a89cf06\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9" Sep 08 04:05:36.594616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:36.594567 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f883400-b336-4ccf-88c0-7fff2a89cf06-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9\" (UID: \"2f883400-b336-4ccf-88c0-7fff2a89cf06\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9" Sep 08 04:05:36.695584 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:36.695562 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6nbj\" (UniqueName: \"kubernetes.io/projected/2f883400-b336-4ccf-88c0-7fff2a89cf06-kube-api-access-x6nbj\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9\" (UID: \"2f883400-b336-4ccf-88c0-7fff2a89cf06\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9" Sep 08 04:05:36.695667 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:36.695590 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f883400-b336-4ccf-88c0-7fff2a89cf06-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9\" (UID: \"2f883400-b336-4ccf-88c0-7fff2a89cf06\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9" Sep 08 04:05:36.695667 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:36.695624 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f883400-b336-4ccf-88c0-7fff2a89cf06-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9\" (UID: \"2f883400-b336-4ccf-88c0-7fff2a89cf06\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9" Sep 08 04:05:36.695906 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:36.695891 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f883400-b336-4ccf-88c0-7fff2a89cf06-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9\" (UID: \"2f883400-b336-4ccf-88c0-7fff2a89cf06\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9" Sep 08 04:05:36.695958 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:36.695929 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f883400-b336-4ccf-88c0-7fff2a89cf06-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9\" (UID: \"2f883400-b336-4ccf-88c0-7fff2a89cf06\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9" Sep 08 04:05:36.706044 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:36.706024 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6nbj\" (UniqueName: \"kubernetes.io/projected/2f883400-b336-4ccf-88c0-7fff2a89cf06-kube-api-access-x6nbj\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9\" (UID: \"2f883400-b336-4ccf-88c0-7fff2a89cf06\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9" Sep 08 04:05:36.858299 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:36.858241 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9" Sep 08 04:05:37.124744 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:37.124606 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9"] Sep 08 04:05:37.127373 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:05:37.127339 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f883400_b336_4ccf_88c0_7fff2a89cf06.slice/crio-c9138801660e9b4940f6627a6a13f117d27daa9f8b4fdb49beb85dd48fc9e39e WatchSource:0}: Error finding container c9138801660e9b4940f6627a6a13f117d27daa9f8b4fdb49beb85dd48fc9e39e: Status 404 returned error can't find the container with id c9138801660e9b4940f6627a6a13f117d27daa9f8b4fdb49beb85dd48fc9e39e Sep 08 04:05:37.129311 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:37.129293 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 04:05:37.157972 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:37.157944 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:05:37.705470 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:37.705443 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9" event={"ID":"2f883400-b336-4ccf-88c0-7fff2a89cf06","Type":"ContainerStarted","Data":"c9138801660e9b4940f6627a6a13f117d27daa9f8b4fdb49beb85dd48fc9e39e"} Sep 08 04:05:42.724254 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:42.724224 2574 generic.go:356] "Generic (PLEG): container finished" podID="2f883400-b336-4ccf-88c0-7fff2a89cf06" containerID="c92e5e0f471189490559173b8beaea51c3eb3bf86b6df5d660afce09185cee2c" exitCode=0 Sep 08 04:05:42.724571 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:42.724307 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9" event={"ID":"2f883400-b336-4ccf-88c0-7fff2a89cf06","Type":"ContainerDied","Data":"c92e5e0f471189490559173b8beaea51c3eb3bf86b6df5d660afce09185cee2c"} Sep 08 04:05:44.733575 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:44.733543 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9" event={"ID":"2f883400-b336-4ccf-88c0-7fff2a89cf06","Type":"ContainerStarted","Data":"efdaa85e1a6b88c9b4ace77dbc9d2a2f17d5462588bbcfa8961f29798eaa3130"} Sep 08 04:05:45.739185 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:45.739151 2574 generic.go:356] "Generic (PLEG): container finished" podID="2f883400-b336-4ccf-88c0-7fff2a89cf06" containerID="efdaa85e1a6b88c9b4ace77dbc9d2a2f17d5462588bbcfa8961f29798eaa3130" exitCode=0 Sep 08 04:05:45.739534 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:45.739239 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9" event={"ID":"2f883400-b336-4ccf-88c0-7fff2a89cf06","Type":"ContainerDied","Data":"efdaa85e1a6b88c9b4ace77dbc9d2a2f17d5462588bbcfa8961f29798eaa3130"} Sep 08 04:05:51.763662 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:51.763631 2574 generic.go:356] "Generic (PLEG): container finished" podID="2f883400-b336-4ccf-88c0-7fff2a89cf06" containerID="eeecafd5975edd63ddc475e74073c3d7633103846669815225e50007d64d8142" exitCode=0 Sep 08 04:05:51.764007 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:51.763696 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9" event={"ID":"2f883400-b336-4ccf-88c0-7fff2a89cf06","Type":"ContainerDied","Data":"eeecafd5975edd63ddc475e74073c3d7633103846669815225e50007d64d8142"} Sep 08 04:05:52.888062 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:52.888039 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9" Sep 08 04:05:53.022710 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:53.022643 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f883400-b336-4ccf-88c0-7fff2a89cf06-bundle\") pod \"2f883400-b336-4ccf-88c0-7fff2a89cf06\" (UID: \"2f883400-b336-4ccf-88c0-7fff2a89cf06\") " Sep 08 04:05:53.022710 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:53.022702 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6nbj\" (UniqueName: \"kubernetes.io/projected/2f883400-b336-4ccf-88c0-7fff2a89cf06-kube-api-access-x6nbj\") pod \"2f883400-b336-4ccf-88c0-7fff2a89cf06\" (UID: \"2f883400-b336-4ccf-88c0-7fff2a89cf06\") " Sep 08 04:05:53.022878 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:53.022785 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f883400-b336-4ccf-88c0-7fff2a89cf06-util\") pod \"2f883400-b336-4ccf-88c0-7fff2a89cf06\" (UID: \"2f883400-b336-4ccf-88c0-7fff2a89cf06\") " Sep 08 04:05:53.023487 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:53.023451 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f883400-b336-4ccf-88c0-7fff2a89cf06-bundle" (OuterVolumeSpecName: "bundle") pod "2f883400-b336-4ccf-88c0-7fff2a89cf06" (UID: "2f883400-b336-4ccf-88c0-7fff2a89cf06"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:05:53.024810 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:53.024781 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f883400-b336-4ccf-88c0-7fff2a89cf06-kube-api-access-x6nbj" (OuterVolumeSpecName: "kube-api-access-x6nbj") pod "2f883400-b336-4ccf-88c0-7fff2a89cf06" (UID: "2f883400-b336-4ccf-88c0-7fff2a89cf06"). InnerVolumeSpecName "kube-api-access-x6nbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:05:53.027327 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:53.027302 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f883400-b336-4ccf-88c0-7fff2a89cf06-util" (OuterVolumeSpecName: "util") pod "2f883400-b336-4ccf-88c0-7fff2a89cf06" (UID: "2f883400-b336-4ccf-88c0-7fff2a89cf06"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:05:53.123183 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:53.123161 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f883400-b336-4ccf-88c0-7fff2a89cf06-util\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:05:53.123183 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:53.123182 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f883400-b336-4ccf-88c0-7fff2a89cf06-bundle\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:05:53.123289 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:53.123190 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6nbj\" (UniqueName: \"kubernetes.io/projected/2f883400-b336-4ccf-88c0-7fff2a89cf06-kube-api-access-x6nbj\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:05:53.772460 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:53.772434 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9" event={"ID":"2f883400-b336-4ccf-88c0-7fff2a89cf06","Type":"ContainerDied","Data":"c9138801660e9b4940f6627a6a13f117d27daa9f8b4fdb49beb85dd48fc9e39e"} Sep 08 04:05:53.772599 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:53.772475 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9138801660e9b4940f6627a6a13f117d27daa9f8b4fdb49beb85dd48fc9e39e" Sep 08 04:05:53.772599 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:53.772444 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rn8t9" Sep 08 04:05:57.552296 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:57.552257 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:05:58.389288 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:58.389252 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:05:59.500048 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:59.500019 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8nql"] Sep 08 04:05:59.500541 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:59.500365 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f883400-b336-4ccf-88c0-7fff2a89cf06" containerName="pull" Sep 08 04:05:59.500541 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:59.500382 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f883400-b336-4ccf-88c0-7fff2a89cf06" containerName="pull" Sep 08 04:05:59.500541 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:59.500396 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f883400-b336-4ccf-88c0-7fff2a89cf06" containerName="util" Sep 08 04:05:59.500541 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:59.500405 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f883400-b336-4ccf-88c0-7fff2a89cf06" containerName="util" Sep 08 04:05:59.500541 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:59.500436 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f883400-b336-4ccf-88c0-7fff2a89cf06" containerName="extract" Sep 08 04:05:59.500541 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:59.500446 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f883400-b336-4ccf-88c0-7fff2a89cf06" containerName="extract" Sep 08 04:05:59.500541 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:59.500523 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f883400-b336-4ccf-88c0-7fff2a89cf06" containerName="extract" Sep 08 04:05:59.503949 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:59.503928 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8nql" Sep 08 04:05:59.509422 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:59.509391 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 08 04:05:59.509539 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:59.509457 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-m8q9n\"" Sep 08 04:05:59.509539 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:59.509466 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 08 04:05:59.522087 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:59.522065 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8nql"] Sep 08 04:05:59.665778 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:59.665751 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a79ef8c6-ebac-4718-b26b-b78eb45f84fa-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-r8nql\" (UID: \"a79ef8c6-ebac-4718-b26b-b78eb45f84fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8nql" Sep 08 04:05:59.665888 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:59.665798 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmb4v\" (UniqueName: \"kubernetes.io/projected/a79ef8c6-ebac-4718-b26b-b78eb45f84fa-kube-api-access-kmb4v\") pod \"cert-manager-operator-controller-manager-55c444bf86-r8nql\" (UID: \"a79ef8c6-ebac-4718-b26b-b78eb45f84fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8nql" Sep 08 04:05:59.766440 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:59.766384 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a79ef8c6-ebac-4718-b26b-b78eb45f84fa-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-r8nql\" (UID: \"a79ef8c6-ebac-4718-b26b-b78eb45f84fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8nql" Sep 08 04:05:59.766440 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:59.766431 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmb4v\" (UniqueName: \"kubernetes.io/projected/a79ef8c6-ebac-4718-b26b-b78eb45f84fa-kube-api-access-kmb4v\") pod \"cert-manager-operator-controller-manager-55c444bf86-r8nql\" (UID: \"a79ef8c6-ebac-4718-b26b-b78eb45f84fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8nql" Sep 08 04:05:59.766726 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:59.766707 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a79ef8c6-ebac-4718-b26b-b78eb45f84fa-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-r8nql\" (UID: \"a79ef8c6-ebac-4718-b26b-b78eb45f84fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8nql" Sep 08 04:05:59.775030 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:59.775013 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmb4v\" (UniqueName: \"kubernetes.io/projected/a79ef8c6-ebac-4718-b26b-b78eb45f84fa-kube-api-access-kmb4v\") pod \"cert-manager-operator-controller-manager-55c444bf86-r8nql\" (UID: \"a79ef8c6-ebac-4718-b26b-b78eb45f84fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8nql" Sep 08 04:05:59.812978 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:05:59.812956 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8nql" Sep 08 04:06:00.093444 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:00.093422 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8nql"] Sep 08 04:06:00.095158 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:06:00.095131 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79ef8c6_ebac_4718_b26b_b78eb45f84fa.slice/crio-a454687169e9f97f12a5d18d1e91ca05bf4f6a891909762387e95c56f03c9b10 WatchSource:0}: Error finding container a454687169e9f97f12a5d18d1e91ca05bf4f6a891909762387e95c56f03c9b10: Status 404 returned error can't find the container with id a454687169e9f97f12a5d18d1e91ca05bf4f6a891909762387e95c56f03c9b10 Sep 08 04:06:00.798850 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:00.798816 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8nql" event={"ID":"a79ef8c6-ebac-4718-b26b-b78eb45f84fa","Type":"ContainerStarted","Data":"a454687169e9f97f12a5d18d1e91ca05bf4f6a891909762387e95c56f03c9b10"} Sep 08 04:06:01.170544 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:01.170456 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:01.270855 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:01.270814 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:01.805742 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:01.805703 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8nql" event={"ID":"a79ef8c6-ebac-4718-b26b-b78eb45f84fa","Type":"ContainerStarted","Data":"f87ac70c33573810e4692201b73e20d6e3e17c2a6adb37218c7b4bcd012dc839"} Sep 08 04:06:01.831033 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:01.830956 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8nql" podStartSLOduration=1.368227136 podStartE2EDuration="2.830941366s" totalImagesPullingTime="1.46271423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:05:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:06:01.829581606 +0000 UTC m=+335.752042366" watchObservedRunningTime="2026-09-08 04:06:01.830941366 +0000 UTC m=+335.753402105" Sep 08 04:06:01.858961 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:01.858933 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:02.261061 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:02.261032 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:02.370360 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:02.370332 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:03.182355 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:03.182324 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl"] Sep 08 04:06:03.184814 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:03.184797 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl" Sep 08 04:06:03.187271 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:03.187246 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 04:06:03.187385 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:03.187306 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-d97f6\"" Sep 08 04:06:03.188182 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:03.188160 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 04:06:03.194218 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:03.194197 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl"] Sep 08 04:06:03.290486 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:03.290459 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl\" (UID: \"1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl" Sep 08 04:06:03.290617 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:03.290522 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl\" (UID: \"1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl" Sep 08 04:06:03.290617 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:03.290559 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlnks\" (UniqueName: \"kubernetes.io/projected/1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f-kube-api-access-mlnks\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl\" (UID: \"1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl" Sep 08 04:06:03.391736 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:03.391709 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl\" (UID: \"1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl" Sep 08 04:06:03.391841 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:03.391751 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl\" (UID: \"1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl" Sep 08 04:06:03.391841 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:03.391786 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlnks\" (UniqueName: \"kubernetes.io/projected/1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f-kube-api-access-mlnks\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl\" (UID: \"1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl" Sep 08 04:06:03.392135 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:03.392114 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl\" (UID: \"1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl" Sep 08 04:06:03.392135 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:03.392129 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl\" (UID: \"1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl" Sep 08 04:06:03.399805 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:03.399777 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlnks\" (UniqueName: \"kubernetes.io/projected/1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f-kube-api-access-mlnks\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl\" (UID: \"1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl" Sep 08 04:06:03.493815 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:03.493763 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl" Sep 08 04:06:03.494455 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:03.494434 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:03.594117 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:03.594087 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:03.771058 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:03.771037 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl"] Sep 08 04:06:03.773030 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:06:03.772992 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e19c47a_8f9c_4ac2_9cd0_ee7e3da3156f.slice/crio-e47d6673749d1ab5720278bed128f477e4ef0ad48a3206096cbe3057eacee5b3 WatchSource:0}: Error finding container e47d6673749d1ab5720278bed128f477e4ef0ad48a3206096cbe3057eacee5b3: Status 404 returned error can't find the container with id e47d6673749d1ab5720278bed128f477e4ef0ad48a3206096cbe3057eacee5b3 Sep 08 04:06:03.814472 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:03.814442 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl" event={"ID":"1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f","Type":"ContainerStarted","Data":"e47d6673749d1ab5720278bed128f477e4ef0ad48a3206096cbe3057eacee5b3"} Sep 08 04:06:04.824446 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:04.824413 2574 generic.go:356] "Generic (PLEG): container finished" podID="1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f" containerID="274db7fd96ea1adbcb3fc0e417548d785ee279b1618e220fbfd7615b8b70d824" exitCode=0 Sep 08 04:06:04.824833 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:04.824465 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl" event={"ID":"1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f","Type":"ContainerDied","Data":"274db7fd96ea1adbcb3fc0e417548d785ee279b1618e220fbfd7615b8b70d824"} Sep 08 04:06:05.501126 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:05.501085 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:05.901613 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:05.901523 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:06.835417 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:06.835394 2574 generic.go:356] "Generic (PLEG): container finished" podID="1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f" containerID="935d5c40d589d16f9fca572379972268876cc32644c1895d5dfbe9dcb14399a6" exitCode=0 Sep 08 04:06:06.835585 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:06.835435 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl" event={"ID":"1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f","Type":"ContainerDied","Data":"935d5c40d589d16f9fca572379972268876cc32644c1895d5dfbe9dcb14399a6"} Sep 08 04:06:07.841381 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:07.841348 2574 generic.go:356] "Generic (PLEG): container finished" podID="1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f" containerID="98bd92f9a2af8ed1f82b39ec04681f4acba8c40159f90b200ee9660356e2f2bd" exitCode=0 Sep 08 04:06:07.841728 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:07.841431 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl" event={"ID":"1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f","Type":"ContainerDied","Data":"98bd92f9a2af8ed1f82b39ec04681f4acba8c40159f90b200ee9660356e2f2bd"} Sep 08 04:06:07.957412 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:07.957389 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:08.021646 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:08.021610 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:08.963686 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:08.963662 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl" Sep 08 04:06:09.130897 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:09.130834 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f-util\") pod \"1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f\" (UID: \"1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f\") " Sep 08 04:06:09.130897 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:09.130887 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f-bundle\") pod \"1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f\" (UID: \"1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f\") " Sep 08 04:06:09.131073 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:09.130931 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlnks\" (UniqueName: \"kubernetes.io/projected/1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f-kube-api-access-mlnks\") pod \"1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f\" (UID: \"1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f\") " Sep 08 04:06:09.131249 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:09.131224 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f-bundle" (OuterVolumeSpecName: "bundle") pod "1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f" (UID: "1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:06:09.132933 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:09.132902 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f-kube-api-access-mlnks" (OuterVolumeSpecName: "kube-api-access-mlnks") pod "1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f" (UID: "1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f"). InnerVolumeSpecName "kube-api-access-mlnks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:06:09.135742 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:09.135717 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f-util" (OuterVolumeSpecName: "util") pod "1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f" (UID: "1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:06:09.232338 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:09.232317 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f-util\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:06:09.232338 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:09.232337 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f-bundle\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:06:09.232453 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:09.232347 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlnks\" (UniqueName: \"kubernetes.io/projected/1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f-kube-api-access-mlnks\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:06:09.855394 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:09.855359 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl" Sep 08 04:06:09.855571 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:09.855363 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk9brl" event={"ID":"1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f","Type":"ContainerDied","Data":"e47d6673749d1ab5720278bed128f477e4ef0ad48a3206096cbe3057eacee5b3"} Sep 08 04:06:09.855571 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:09.855528 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e47d6673749d1ab5720278bed128f477e4ef0ad48a3206096cbe3057eacee5b3" Sep 08 04:06:13.966372 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:13.966338 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:14.067857 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:14.067820 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:14.166004 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:14.165975 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:16.181938 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:16.181905 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9w4v5"] Sep 08 04:06:16.182283 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:16.182142 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f" containerName="util" Sep 08 04:06:16.182283 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:16.182153 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f" containerName="util" Sep 08 04:06:16.182283 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:16.182162 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f" containerName="extract" Sep 08 04:06:16.182283 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:16.182168 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f" containerName="extract" Sep 08 04:06:16.182283 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:16.182182 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f" containerName="pull" Sep 08 04:06:16.182283 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:16.182187 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f" containerName="pull" Sep 08 04:06:16.182283 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:16.182234 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e19c47a-8f9c-4ac2-9cd0-ee7e3da3156f" containerName="extract" Sep 08 04:06:16.184361 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:16.184345 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9w4v5" Sep 08 04:06:16.186636 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:16.186614 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 08 04:06:16.186746 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:16.186614 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 08 04:06:16.187585 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:16.187568 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-q2lth\"" Sep 08 04:06:16.192798 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:16.192778 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9w4v5"] Sep 08 04:06:16.273007 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:16.272986 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e10c809c-a846-48c6-85a6-ca6554bcb5fe-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9w4v5\" (UID: \"e10c809c-a846-48c6-85a6-ca6554bcb5fe\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9w4v5" Sep 08 04:06:16.273126 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:16.273032 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsmxd\" (UniqueName: \"kubernetes.io/projected/e10c809c-a846-48c6-85a6-ca6554bcb5fe-kube-api-access-zsmxd\") pod \"openshift-lws-operator-fd8ccff4c-9w4v5\" (UID: \"e10c809c-a846-48c6-85a6-ca6554bcb5fe\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9w4v5" Sep 08 04:06:16.373971 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:16.373946 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e10c809c-a846-48c6-85a6-ca6554bcb5fe-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9w4v5\" (UID: \"e10c809c-a846-48c6-85a6-ca6554bcb5fe\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9w4v5" Sep 08 04:06:16.374066 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:16.373985 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsmxd\" (UniqueName: \"kubernetes.io/projected/e10c809c-a846-48c6-85a6-ca6554bcb5fe-kube-api-access-zsmxd\") pod \"openshift-lws-operator-fd8ccff4c-9w4v5\" (UID: \"e10c809c-a846-48c6-85a6-ca6554bcb5fe\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9w4v5" Sep 08 04:06:16.374266 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:16.374247 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e10c809c-a846-48c6-85a6-ca6554bcb5fe-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9w4v5\" (UID: \"e10c809c-a846-48c6-85a6-ca6554bcb5fe\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9w4v5" Sep 08 04:06:16.381926 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:16.381902 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsmxd\" (UniqueName: \"kubernetes.io/projected/e10c809c-a846-48c6-85a6-ca6554bcb5fe-kube-api-access-zsmxd\") pod \"openshift-lws-operator-fd8ccff4c-9w4v5\" (UID: \"e10c809c-a846-48c6-85a6-ca6554bcb5fe\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9w4v5" Sep 08 04:06:16.505045 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:16.505026 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9w4v5" Sep 08 04:06:16.771126 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:16.771105 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9w4v5"] Sep 08 04:06:16.773129 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:06:16.773086 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode10c809c_a846_48c6_85a6_ca6554bcb5fe.slice/crio-955b1ddde8d7ef1681a85d6d2cc09d4f5e971835f67ae9e7e21fb52bd02f94e0 WatchSource:0}: Error finding container 955b1ddde8d7ef1681a85d6d2cc09d4f5e971835f67ae9e7e21fb52bd02f94e0: Status 404 returned error can't find the container with id 955b1ddde8d7ef1681a85d6d2cc09d4f5e971835f67ae9e7e21fb52bd02f94e0 Sep 08 04:06:16.882009 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:16.881984 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9w4v5" event={"ID":"e10c809c-a846-48c6-85a6-ca6554bcb5fe","Type":"ContainerStarted","Data":"955b1ddde8d7ef1681a85d6d2cc09d4f5e971835f67ae9e7e21fb52bd02f94e0"} Sep 08 04:06:18.895070 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:18.894991 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9w4v5" event={"ID":"e10c809c-a846-48c6-85a6-ca6554bcb5fe","Type":"ContainerStarted","Data":"ca1b0c6c1f345b25b50a425312cf9508e27247c360d825580ef898f721aaa726"} Sep 08 04:06:18.913671 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:18.913625 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9w4v5" podStartSLOduration=1.403875475 podStartE2EDuration="2.913611559s" totalImagesPullingTime="1.509736084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:06:18.913059256 +0000 UTC m=+352.835519997" watchObservedRunningTime="2026-09-08 04:06:18.913611559 +0000 UTC m=+352.836072300" Sep 08 04:06:19.703863 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:19.703827 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:19.811604 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:19.811545 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:21.790928 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:21.790901 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:23.967546 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:23.967515 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz"] Sep 08 04:06:23.970979 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:23.970963 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz" Sep 08 04:06:23.973390 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:23.973367 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 04:06:23.974311 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:23.974295 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-d97f6\"" Sep 08 04:06:23.974407 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:23.974301 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 04:06:23.978256 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:23.978236 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz"] Sep 08 04:06:24.125230 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:24.125207 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9810eea7-2113-47c3-8b87-b414e5306aff-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz\" (UID: \"9810eea7-2113-47c3-8b87-b414e5306aff\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz" Sep 08 04:06:24.125316 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:24.125250 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgmmz\" (UniqueName: \"kubernetes.io/projected/9810eea7-2113-47c3-8b87-b414e5306aff-kube-api-access-hgmmz\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz\" (UID: \"9810eea7-2113-47c3-8b87-b414e5306aff\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz" Sep 08 04:06:24.125361 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:24.125310 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9810eea7-2113-47c3-8b87-b414e5306aff-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz\" (UID: \"9810eea7-2113-47c3-8b87-b414e5306aff\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz" Sep 08 04:06:24.226434 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:24.226383 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9810eea7-2113-47c3-8b87-b414e5306aff-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz\" (UID: \"9810eea7-2113-47c3-8b87-b414e5306aff\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz" Sep 08 04:06:24.226434 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:24.226417 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9810eea7-2113-47c3-8b87-b414e5306aff-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz\" (UID: \"9810eea7-2113-47c3-8b87-b414e5306aff\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz" Sep 08 04:06:24.226568 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:24.226450 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgmmz\" (UniqueName: \"kubernetes.io/projected/9810eea7-2113-47c3-8b87-b414e5306aff-kube-api-access-hgmmz\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz\" (UID: \"9810eea7-2113-47c3-8b87-b414e5306aff\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz" Sep 08 04:06:24.226758 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:24.226740 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9810eea7-2113-47c3-8b87-b414e5306aff-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz\" (UID: \"9810eea7-2113-47c3-8b87-b414e5306aff\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz" Sep 08 04:06:24.226800 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:24.226781 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9810eea7-2113-47c3-8b87-b414e5306aff-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz\" (UID: \"9810eea7-2113-47c3-8b87-b414e5306aff\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz" Sep 08 04:06:24.234259 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:24.234238 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgmmz\" (UniqueName: \"kubernetes.io/projected/9810eea7-2113-47c3-8b87-b414e5306aff-kube-api-access-hgmmz\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz\" (UID: \"9810eea7-2113-47c3-8b87-b414e5306aff\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz" Sep 08 04:06:24.281140 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:24.281117 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz" Sep 08 04:06:24.560555 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:24.560530 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz"] Sep 08 04:06:24.562531 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:06:24.562484 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9810eea7_2113_47c3_8b87_b414e5306aff.slice/crio-c4fb147cdbce4f34c4ad6243d8c1650b66dc73f0d5727702c00abb09849b5892 WatchSource:0}: Error finding container c4fb147cdbce4f34c4ad6243d8c1650b66dc73f0d5727702c00abb09849b5892: Status 404 returned error can't find the container with id c4fb147cdbce4f34c4ad6243d8c1650b66dc73f0d5727702c00abb09849b5892 Sep 08 04:06:24.919150 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:24.919083 2574 generic.go:356] "Generic (PLEG): container finished" podID="9810eea7-2113-47c3-8b87-b414e5306aff" containerID="d07296cb0317df631ace391e0f73524864604484a9f65b231f4a630b43f85210" exitCode=0 Sep 08 04:06:24.919150 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:24.919141 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz" event={"ID":"9810eea7-2113-47c3-8b87-b414e5306aff","Type":"ContainerDied","Data":"d07296cb0317df631ace391e0f73524864604484a9f65b231f4a630b43f85210"} Sep 08 04:06:24.919313 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:24.919190 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz" event={"ID":"9810eea7-2113-47c3-8b87-b414e5306aff","Type":"ContainerStarted","Data":"c4fb147cdbce4f34c4ad6243d8c1650b66dc73f0d5727702c00abb09849b5892"} Sep 08 04:06:26.928180 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:26.928149 2574 generic.go:356] "Generic (PLEG): container finished" podID="9810eea7-2113-47c3-8b87-b414e5306aff" containerID="d16aac204938ab6054a68deec35e589e6e60f8f44ca086622e36b11be5db3f8d" exitCode=0 Sep 08 04:06:26.928544 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:26.928195 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz" event={"ID":"9810eea7-2113-47c3-8b87-b414e5306aff","Type":"ContainerDied","Data":"d16aac204938ab6054a68deec35e589e6e60f8f44ca086622e36b11be5db3f8d"} Sep 08 04:06:27.633990 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:27.633960 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:27.694675 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:27.694655 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:27.935017 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:27.934956 2574 generic.go:356] "Generic (PLEG): container finished" podID="9810eea7-2113-47c3-8b87-b414e5306aff" containerID="08a504929bc4a1135508ae100f98514c12575cc1389c5eaf1ccc49ebef481a5c" exitCode=0 Sep 08 04:06:27.935293 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:27.935013 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz" event={"ID":"9810eea7-2113-47c3-8b87-b414e5306aff","Type":"ContainerDied","Data":"08a504929bc4a1135508ae100f98514c12575cc1389c5eaf1ccc49ebef481a5c"} Sep 08 04:06:29.054385 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:29.054362 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz" Sep 08 04:06:29.159788 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:29.159757 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9810eea7-2113-47c3-8b87-b414e5306aff-util\") pod \"9810eea7-2113-47c3-8b87-b414e5306aff\" (UID: \"9810eea7-2113-47c3-8b87-b414e5306aff\") " Sep 08 04:06:29.159917 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:29.159801 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgmmz\" (UniqueName: \"kubernetes.io/projected/9810eea7-2113-47c3-8b87-b414e5306aff-kube-api-access-hgmmz\") pod \"9810eea7-2113-47c3-8b87-b414e5306aff\" (UID: \"9810eea7-2113-47c3-8b87-b414e5306aff\") " Sep 08 04:06:29.159917 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:29.159838 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9810eea7-2113-47c3-8b87-b414e5306aff-bundle\") pod \"9810eea7-2113-47c3-8b87-b414e5306aff\" (UID: \"9810eea7-2113-47c3-8b87-b414e5306aff\") " Sep 08 04:06:29.160614 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:29.160588 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9810eea7-2113-47c3-8b87-b414e5306aff-bundle" (OuterVolumeSpecName: "bundle") pod "9810eea7-2113-47c3-8b87-b414e5306aff" (UID: "9810eea7-2113-47c3-8b87-b414e5306aff"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:06:29.161758 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:29.161723 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9810eea7-2113-47c3-8b87-b414e5306aff-kube-api-access-hgmmz" (OuterVolumeSpecName: "kube-api-access-hgmmz") pod "9810eea7-2113-47c3-8b87-b414e5306aff" (UID: "9810eea7-2113-47c3-8b87-b414e5306aff"). InnerVolumeSpecName "kube-api-access-hgmmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:06:29.165544 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:29.165522 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9810eea7-2113-47c3-8b87-b414e5306aff-util" (OuterVolumeSpecName: "util") pod "9810eea7-2113-47c3-8b87-b414e5306aff" (UID: "9810eea7-2113-47c3-8b87-b414e5306aff"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:06:29.261085 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:29.261059 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9810eea7-2113-47c3-8b87-b414e5306aff-util\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:06:29.261085 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:29.261086 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgmmz\" (UniqueName: \"kubernetes.io/projected/9810eea7-2113-47c3-8b87-b414e5306aff-kube-api-access-hgmmz\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:06:29.261214 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:29.261095 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9810eea7-2113-47c3-8b87-b414e5306aff-bundle\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:06:29.945867 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:29.945844 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz" Sep 08 04:06:29.945990 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:29.945862 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs2xbz" event={"ID":"9810eea7-2113-47c3-8b87-b414e5306aff","Type":"ContainerDied","Data":"c4fb147cdbce4f34c4ad6243d8c1650b66dc73f0d5727702c00abb09849b5892"} Sep 08 04:06:29.945990 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:29.945898 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4fb147cdbce4f34c4ad6243d8c1650b66dc73f0d5727702c00abb09849b5892" Sep 08 04:06:34.586806 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:34.586772 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn"] Sep 08 04:06:34.587861 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:34.587814 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9810eea7-2113-47c3-8b87-b414e5306aff" containerName="extract" Sep 08 04:06:34.588051 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:34.588036 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9810eea7-2113-47c3-8b87-b414e5306aff" containerName="extract" Sep 08 04:06:34.588178 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:34.588167 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9810eea7-2113-47c3-8b87-b414e5306aff" containerName="pull" Sep 08 04:06:34.588272 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:34.588262 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9810eea7-2113-47c3-8b87-b414e5306aff" containerName="pull" Sep 08 04:06:34.588409 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:34.588399 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9810eea7-2113-47c3-8b87-b414e5306aff" containerName="util" Sep 08 04:06:34.588497 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:34.588488 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9810eea7-2113-47c3-8b87-b414e5306aff" containerName="util" Sep 08 04:06:34.588756 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:34.588743 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9810eea7-2113-47c3-8b87-b414e5306aff" containerName="extract" Sep 08 04:06:34.595792 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:34.595770 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn" Sep 08 04:06:34.597019 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:34.596996 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn"] Sep 08 04:06:34.598616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:34.598590 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 04:06:34.598616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:34.598612 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 04:06:34.598760 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:34.598625 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-d97f6\"" Sep 08 04:06:34.694250 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:34.694225 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzt6m\" (UniqueName: \"kubernetes.io/projected/43cbe51d-3616-4262-aaab-afb9a884e0b0-kube-api-access-vzt6m\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn\" (UID: \"43cbe51d-3616-4262-aaab-afb9a884e0b0\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn" Sep 08 04:06:34.694350 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:34.694297 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43cbe51d-3616-4262-aaab-afb9a884e0b0-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn\" (UID: \"43cbe51d-3616-4262-aaab-afb9a884e0b0\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn" Sep 08 04:06:34.694350 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:34.694342 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43cbe51d-3616-4262-aaab-afb9a884e0b0-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn\" (UID: \"43cbe51d-3616-4262-aaab-afb9a884e0b0\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn" Sep 08 04:06:34.795348 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:34.795324 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43cbe51d-3616-4262-aaab-afb9a884e0b0-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn\" (UID: \"43cbe51d-3616-4262-aaab-afb9a884e0b0\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn" Sep 08 04:06:34.795450 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:34.795359 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43cbe51d-3616-4262-aaab-afb9a884e0b0-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn\" (UID: \"43cbe51d-3616-4262-aaab-afb9a884e0b0\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn" Sep 08 04:06:34.795450 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:34.795387 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzt6m\" (UniqueName: \"kubernetes.io/projected/43cbe51d-3616-4262-aaab-afb9a884e0b0-kube-api-access-vzt6m\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn\" (UID: \"43cbe51d-3616-4262-aaab-afb9a884e0b0\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn" Sep 08 04:06:34.795724 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:34.795706 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43cbe51d-3616-4262-aaab-afb9a884e0b0-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn\" (UID: \"43cbe51d-3616-4262-aaab-afb9a884e0b0\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn" Sep 08 04:06:34.795790 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:34.795745 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43cbe51d-3616-4262-aaab-afb9a884e0b0-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn\" (UID: \"43cbe51d-3616-4262-aaab-afb9a884e0b0\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn" Sep 08 04:06:34.806463 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:34.806440 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzt6m\" (UniqueName: \"kubernetes.io/projected/43cbe51d-3616-4262-aaab-afb9a884e0b0-kube-api-access-vzt6m\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn\" (UID: \"43cbe51d-3616-4262-aaab-afb9a884e0b0\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn" Sep 08 04:06:34.905134 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:34.905082 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn" Sep 08 04:06:35.177416 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:35.177362 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn"] Sep 08 04:06:35.179380 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:06:35.179354 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43cbe51d_3616_4262_aaab_afb9a884e0b0.slice/crio-d0c635a4a00b80e276f6636381fca258d2712555e92e4b794542f10ba44c9343 WatchSource:0}: Error finding container d0c635a4a00b80e276f6636381fca258d2712555e92e4b794542f10ba44c9343: Status 404 returned error can't find the container with id d0c635a4a00b80e276f6636381fca258d2712555e92e4b794542f10ba44c9343 Sep 08 04:06:35.795157 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:35.795130 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:35.895070 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:35.895038 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:35.972359 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:35.972338 2574 generic.go:356] "Generic (PLEG): container finished" podID="43cbe51d-3616-4262-aaab-afb9a884e0b0" containerID="ad30cb6829205075d06ad4d2f4994b3556450a8c55f2ec2eede572819035d12b" exitCode=0 Sep 08 04:06:35.972485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:35.972384 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn" event={"ID":"43cbe51d-3616-4262-aaab-afb9a884e0b0","Type":"ContainerDied","Data":"ad30cb6829205075d06ad4d2f4994b3556450a8c55f2ec2eede572819035d12b"} Sep 08 04:06:35.972485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:35.972420 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn" event={"ID":"43cbe51d-3616-4262-aaab-afb9a884e0b0","Type":"ContainerStarted","Data":"d0c635a4a00b80e276f6636381fca258d2712555e92e4b794542f10ba44c9343"} Sep 08 04:06:36.455148 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:36.455115 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:36.846143 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:36.846117 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:36.950575 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:36.950550 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:36.978385 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:36.978357 2574 generic.go:356] "Generic (PLEG): container finished" podID="43cbe51d-3616-4262-aaab-afb9a884e0b0" containerID="d0ed199f738a1dfeb091d876ae3ce44fac3b50a6aa206d5dcc311e98bc4cdf7a" exitCode=0 Sep 08 04:06:36.978480 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:36.978441 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn" event={"ID":"43cbe51d-3616-4262-aaab-afb9a884e0b0","Type":"ContainerDied","Data":"d0ed199f738a1dfeb091d876ae3ce44fac3b50a6aa206d5dcc311e98bc4cdf7a"} Sep 08 04:06:37.001741 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.001721 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d5d59f488-6t42t"] Sep 08 04:06:37.005926 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.005909 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-6t42t" Sep 08 04:06:37.008428 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.008409 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 08 04:06:37.008525 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.008409 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jkdxt\"" Sep 08 04:06:37.008663 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.008631 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 08 04:06:37.008696 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.008688 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 08 04:06:37.009319 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.009305 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 08 04:06:37.032258 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.032223 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d5d59f488-6t42t"] Sep 08 04:06:37.110711 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.110643 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f718da3d-1d9b-489d-87fe-a241d6f748c1-manifests\") pod \"opendatahub-operator-controller-manager-5d5d59f488-6t42t\" (UID: \"f718da3d-1d9b-489d-87fe-a241d6f748c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-6t42t" Sep 08 04:06:37.110711 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.110671 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78wlc\" (UniqueName: \"kubernetes.io/projected/f718da3d-1d9b-489d-87fe-a241d6f748c1-kube-api-access-78wlc\") pod \"opendatahub-operator-controller-manager-5d5d59f488-6t42t\" (UID: \"f718da3d-1d9b-489d-87fe-a241d6f748c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-6t42t" Sep 08 04:06:37.110711 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.110700 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f718da3d-1d9b-489d-87fe-a241d6f748c1-tmp\") pod \"opendatahub-operator-controller-manager-5d5d59f488-6t42t\" (UID: \"f718da3d-1d9b-489d-87fe-a241d6f748c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-6t42t" Sep 08 04:06:37.110916 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.110802 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f718da3d-1d9b-489d-87fe-a241d6f748c1-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d5d59f488-6t42t\" (UID: \"f718da3d-1d9b-489d-87fe-a241d6f748c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-6t42t" Sep 08 04:06:37.110916 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.110830 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f718da3d-1d9b-489d-87fe-a241d6f748c1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d5d59f488-6t42t\" (UID: \"f718da3d-1d9b-489d-87fe-a241d6f748c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-6t42t" Sep 08 04:06:37.125314 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.125291 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:37.211302 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.211278 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f718da3d-1d9b-489d-87fe-a241d6f748c1-tmp\") pod \"opendatahub-operator-controller-manager-5d5d59f488-6t42t\" (UID: \"f718da3d-1d9b-489d-87fe-a241d6f748c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-6t42t" Sep 08 04:06:37.211409 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.211318 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f718da3d-1d9b-489d-87fe-a241d6f748c1-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d5d59f488-6t42t\" (UID: \"f718da3d-1d9b-489d-87fe-a241d6f748c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-6t42t" Sep 08 04:06:37.211409 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.211358 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f718da3d-1d9b-489d-87fe-a241d6f748c1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d5d59f488-6t42t\" (UID: \"f718da3d-1d9b-489d-87fe-a241d6f748c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-6t42t" Sep 08 04:06:37.211585 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.211412 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f718da3d-1d9b-489d-87fe-a241d6f748c1-manifests\") pod \"opendatahub-operator-controller-manager-5d5d59f488-6t42t\" (UID: \"f718da3d-1d9b-489d-87fe-a241d6f748c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-6t42t" Sep 08 04:06:37.211585 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.211441 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78wlc\" (UniqueName: \"kubernetes.io/projected/f718da3d-1d9b-489d-87fe-a241d6f748c1-kube-api-access-78wlc\") pod \"opendatahub-operator-controller-manager-5d5d59f488-6t42t\" (UID: \"f718da3d-1d9b-489d-87fe-a241d6f748c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-6t42t" Sep 08 04:06:37.211768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.211747 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f718da3d-1d9b-489d-87fe-a241d6f748c1-manifests\") pod \"opendatahub-operator-controller-manager-5d5d59f488-6t42t\" (UID: \"f718da3d-1d9b-489d-87fe-a241d6f748c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-6t42t" Sep 08 04:06:37.213365 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.213348 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f718da3d-1d9b-489d-87fe-a241d6f748c1-tmp\") pod \"opendatahub-operator-controller-manager-5d5d59f488-6t42t\" (UID: \"f718da3d-1d9b-489d-87fe-a241d6f748c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-6t42t" Sep 08 04:06:37.213601 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.213583 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f718da3d-1d9b-489d-87fe-a241d6f748c1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d5d59f488-6t42t\" (UID: \"f718da3d-1d9b-489d-87fe-a241d6f748c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-6t42t" Sep 08 04:06:37.213748 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.213729 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f718da3d-1d9b-489d-87fe-a241d6f748c1-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d5d59f488-6t42t\" (UID: \"f718da3d-1d9b-489d-87fe-a241d6f748c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-6t42t" Sep 08 04:06:37.218721 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.218696 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:37.219849 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.219817 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78wlc\" (UniqueName: \"kubernetes.io/projected/f718da3d-1d9b-489d-87fe-a241d6f748c1-kube-api-access-78wlc\") pod \"opendatahub-operator-controller-manager-5d5d59f488-6t42t\" (UID: \"f718da3d-1d9b-489d-87fe-a241d6f748c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-6t42t" Sep 08 04:06:37.319299 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.319275 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-6t42t" Sep 08 04:06:37.328122 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.328090 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:37.422472 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.422441 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:37.520221 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.520194 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:37.594907 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.593647 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d5d59f488-6t42t"] Sep 08 04:06:37.596607 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:06:37.596575 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf718da3d_1d9b_489d_87fe_a241d6f748c1.slice/crio-8edc862ae614d29a552dd1393fae4c4d8cc8236f9fce332afc83011a66787b6c WatchSource:0}: Error finding container 8edc862ae614d29a552dd1393fae4c4d8cc8236f9fce332afc83011a66787b6c: Status 404 returned error can't find the container with id 8edc862ae614d29a552dd1393fae4c4d8cc8236f9fce332afc83011a66787b6c Sep 08 04:06:37.598808 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.598783 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:37.684909 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.684846 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:37.985618 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.985559 2574 generic.go:356] "Generic (PLEG): container finished" podID="43cbe51d-3616-4262-aaab-afb9a884e0b0" containerID="a668e20894d632adc8fec643e2034f61058bc2857bde874ee87d082a812f1b3a" exitCode=0 Sep 08 04:06:37.985969 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.985637 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn" event={"ID":"43cbe51d-3616-4262-aaab-afb9a884e0b0","Type":"ContainerDied","Data":"a668e20894d632adc8fec643e2034f61058bc2857bde874ee87d082a812f1b3a"} Sep 08 04:06:37.986978 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:37.986961 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-6t42t" event={"ID":"f718da3d-1d9b-489d-87fe-a241d6f748c1","Type":"ContainerStarted","Data":"8edc862ae614d29a552dd1393fae4c4d8cc8236f9fce332afc83011a66787b6c"} Sep 08 04:06:40.504043 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:40.504014 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn" Sep 08 04:06:40.638184 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:40.638157 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43cbe51d-3616-4262-aaab-afb9a884e0b0-bundle\") pod \"43cbe51d-3616-4262-aaab-afb9a884e0b0\" (UID: \"43cbe51d-3616-4262-aaab-afb9a884e0b0\") " Sep 08 04:06:40.638292 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:40.638196 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43cbe51d-3616-4262-aaab-afb9a884e0b0-util\") pod \"43cbe51d-3616-4262-aaab-afb9a884e0b0\" (UID: \"43cbe51d-3616-4262-aaab-afb9a884e0b0\") " Sep 08 04:06:40.638292 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:40.638261 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzt6m\" (UniqueName: \"kubernetes.io/projected/43cbe51d-3616-4262-aaab-afb9a884e0b0-kube-api-access-vzt6m\") pod \"43cbe51d-3616-4262-aaab-afb9a884e0b0\" (UID: \"43cbe51d-3616-4262-aaab-afb9a884e0b0\") " Sep 08 04:06:40.638829 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:40.638800 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cbe51d-3616-4262-aaab-afb9a884e0b0-bundle" (OuterVolumeSpecName: "bundle") pod "43cbe51d-3616-4262-aaab-afb9a884e0b0" (UID: "43cbe51d-3616-4262-aaab-afb9a884e0b0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:06:40.640000 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:40.639977 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43cbe51d-3616-4262-aaab-afb9a884e0b0-kube-api-access-vzt6m" (OuterVolumeSpecName: "kube-api-access-vzt6m") pod "43cbe51d-3616-4262-aaab-afb9a884e0b0" (UID: "43cbe51d-3616-4262-aaab-afb9a884e0b0"). InnerVolumeSpecName "kube-api-access-vzt6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:06:40.645652 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:40.645625 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cbe51d-3616-4262-aaab-afb9a884e0b0-util" (OuterVolumeSpecName: "util") pod "43cbe51d-3616-4262-aaab-afb9a884e0b0" (UID: "43cbe51d-3616-4262-aaab-afb9a884e0b0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:06:40.739187 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:40.739138 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzt6m\" (UniqueName: \"kubernetes.io/projected/43cbe51d-3616-4262-aaab-afb9a884e0b0-kube-api-access-vzt6m\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:06:40.739187 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:40.739161 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43cbe51d-3616-4262-aaab-afb9a884e0b0-bundle\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:06:40.739187 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:40.739170 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43cbe51d-3616-4262-aaab-afb9a884e0b0-util\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:06:41.003912 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:41.003886 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-6t42t" event={"ID":"f718da3d-1d9b-489d-87fe-a241d6f748c1","Type":"ContainerStarted","Data":"d38bc26c8cef10e8e0c8d7e82dd4226e8dbc35a1d713f864a640ff888689c7bf"} Sep 08 04:06:41.005705 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:41.005681 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn" event={"ID":"43cbe51d-3616-4262-aaab-afb9a884e0b0","Type":"ContainerDied","Data":"d0c635a4a00b80e276f6636381fca258d2712555e92e4b794542f10ba44c9343"} Sep 08 04:06:41.005791 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:41.005711 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0s6mrn" Sep 08 04:06:41.005791 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:41.005725 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0c635a4a00b80e276f6636381fca258d2712555e92e4b794542f10ba44c9343" Sep 08 04:06:42.011622 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:42.011590 2574 generic.go:356] "Generic (PLEG): container finished" podID="f718da3d-1d9b-489d-87fe-a241d6f748c1" containerID="d38bc26c8cef10e8e0c8d7e82dd4226e8dbc35a1d713f864a640ff888689c7bf" exitCode=0 Sep 08 04:06:42.011971 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:42.011676 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-6t42t" event={"ID":"f718da3d-1d9b-489d-87fe-a241d6f748c1","Type":"ContainerDied","Data":"d38bc26c8cef10e8e0c8d7e82dd4226e8dbc35a1d713f864a640ff888689c7bf"} Sep 08 04:06:43.022625 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:43.022590 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-6t42t" event={"ID":"f718da3d-1d9b-489d-87fe-a241d6f748c1","Type":"ContainerStarted","Data":"5765d0d09df3b906f3052b88ba881f989d19f14494ddab58eaa5d4dc35effa7c"} Sep 08 04:06:43.023092 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:43.022770 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-6t42t" Sep 08 04:06:43.046715 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:43.046665 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-6t42t" podStartSLOduration=2.8023624480000002 podStartE2EDuration="7.046651132s" totalImagesPullingTime="3.241425413s" totalInitContainerRuntime="1.002863271s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:06:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:06:43.045367306 +0000 UTC m=+376.967828105" watchObservedRunningTime="2026-09-08 04:06:43.046651132 +0000 UTC m=+376.969111872" Sep 08 04:06:53.123698 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:53.123668 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:53.196033 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:53.195998 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:54.028808 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:54.028779 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5d5d59f488-6t42t" Sep 08 04:06:55.158063 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.158030 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-58d7df9668-hgqgf"] Sep 08 04:06:55.158404 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.158310 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43cbe51d-3616-4262-aaab-afb9a884e0b0" containerName="pull" Sep 08 04:06:55.158404 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.158322 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="43cbe51d-3616-4262-aaab-afb9a884e0b0" containerName="pull" Sep 08 04:06:55.158404 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.158330 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43cbe51d-3616-4262-aaab-afb9a884e0b0" containerName="util" Sep 08 04:06:55.158404 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.158336 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="43cbe51d-3616-4262-aaab-afb9a884e0b0" containerName="util" Sep 08 04:06:55.158404 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.158347 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43cbe51d-3616-4262-aaab-afb9a884e0b0" containerName="extract" Sep 08 04:06:55.158404 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.158355 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="43cbe51d-3616-4262-aaab-afb9a884e0b0" containerName="extract" Sep 08 04:06:55.158404 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.158402 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="43cbe51d-3616-4262-aaab-afb9a884e0b0" containerName="extract" Sep 08 04:06:55.161309 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.161289 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58d7df9668-hgqgf" Sep 08 04:06:55.165005 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.164979 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 08 04:06:55.165136 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.164983 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 08 04:06:55.165136 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.165012 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-55m7s\"" Sep 08 04:06:55.165136 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.165016 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 08 04:06:55.171220 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.171197 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58d7df9668-hgqgf"] Sep 08 04:06:55.238802 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.238779 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c5b842e7-f15f-4453-bb7e-8191c1951874-cert\") pod \"lws-controller-manager-58d7df9668-hgqgf\" (UID: \"c5b842e7-f15f-4453-bb7e-8191c1951874\") " pod="openshift-lws-operator/lws-controller-manager-58d7df9668-hgqgf" Sep 08 04:06:55.238897 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.238815 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c5b842e7-f15f-4453-bb7e-8191c1951874-manager-config\") pod \"lws-controller-manager-58d7df9668-hgqgf\" (UID: \"c5b842e7-f15f-4453-bb7e-8191c1951874\") " pod="openshift-lws-operator/lws-controller-manager-58d7df9668-hgqgf" Sep 08 04:06:55.238897 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.238836 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6r8p\" (UniqueName: \"kubernetes.io/projected/c5b842e7-f15f-4453-bb7e-8191c1951874-kube-api-access-w6r8p\") pod \"lws-controller-manager-58d7df9668-hgqgf\" (UID: \"c5b842e7-f15f-4453-bb7e-8191c1951874\") " pod="openshift-lws-operator/lws-controller-manager-58d7df9668-hgqgf" Sep 08 04:06:55.238992 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.238914 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5b842e7-f15f-4453-bb7e-8191c1951874-metrics-cert\") pod \"lws-controller-manager-58d7df9668-hgqgf\" (UID: \"c5b842e7-f15f-4453-bb7e-8191c1951874\") " pod="openshift-lws-operator/lws-controller-manager-58d7df9668-hgqgf" Sep 08 04:06:55.339542 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.339521 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5b842e7-f15f-4453-bb7e-8191c1951874-metrics-cert\") pod \"lws-controller-manager-58d7df9668-hgqgf\" (UID: \"c5b842e7-f15f-4453-bb7e-8191c1951874\") " pod="openshift-lws-operator/lws-controller-manager-58d7df9668-hgqgf" Sep 08 04:06:55.339633 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.339566 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c5b842e7-f15f-4453-bb7e-8191c1951874-cert\") pod \"lws-controller-manager-58d7df9668-hgqgf\" (UID: \"c5b842e7-f15f-4453-bb7e-8191c1951874\") " pod="openshift-lws-operator/lws-controller-manager-58d7df9668-hgqgf" Sep 08 04:06:55.339633 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.339593 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c5b842e7-f15f-4453-bb7e-8191c1951874-manager-config\") pod \"lws-controller-manager-58d7df9668-hgqgf\" (UID: \"c5b842e7-f15f-4453-bb7e-8191c1951874\") " pod="openshift-lws-operator/lws-controller-manager-58d7df9668-hgqgf" Sep 08 04:06:55.339633 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.339615 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6r8p\" (UniqueName: \"kubernetes.io/projected/c5b842e7-f15f-4453-bb7e-8191c1951874-kube-api-access-w6r8p\") pod \"lws-controller-manager-58d7df9668-hgqgf\" (UID: \"c5b842e7-f15f-4453-bb7e-8191c1951874\") " pod="openshift-lws-operator/lws-controller-manager-58d7df9668-hgqgf" Sep 08 04:06:55.340294 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.340266 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c5b842e7-f15f-4453-bb7e-8191c1951874-manager-config\") pod \"lws-controller-manager-58d7df9668-hgqgf\" (UID: \"c5b842e7-f15f-4453-bb7e-8191c1951874\") " pod="openshift-lws-operator/lws-controller-manager-58d7df9668-hgqgf" Sep 08 04:06:55.342015 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.341995 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c5b842e7-f15f-4453-bb7e-8191c1951874-cert\") pod \"lws-controller-manager-58d7df9668-hgqgf\" (UID: \"c5b842e7-f15f-4453-bb7e-8191c1951874\") " pod="openshift-lws-operator/lws-controller-manager-58d7df9668-hgqgf" Sep 08 04:06:55.342096 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.342041 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5b842e7-f15f-4453-bb7e-8191c1951874-metrics-cert\") pod \"lws-controller-manager-58d7df9668-hgqgf\" (UID: \"c5b842e7-f15f-4453-bb7e-8191c1951874\") " pod="openshift-lws-operator/lws-controller-manager-58d7df9668-hgqgf" Sep 08 04:06:55.348566 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.347979 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6r8p\" (UniqueName: \"kubernetes.io/projected/c5b842e7-f15f-4453-bb7e-8191c1951874-kube-api-access-w6r8p\") pod \"lws-controller-manager-58d7df9668-hgqgf\" (UID: \"c5b842e7-f15f-4453-bb7e-8191c1951874\") " pod="openshift-lws-operator/lws-controller-manager-58d7df9668-hgqgf" Sep 08 04:06:55.470942 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.470881 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58d7df9668-hgqgf" Sep 08 04:06:55.750879 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:55.750853 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58d7df9668-hgqgf"] Sep 08 04:06:55.753357 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:06:55.753324 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b842e7_f15f_4453_bb7e_8191c1951874.slice/crio-ece1cfe70b85e25121862806093a5b04e6084c10157337f08612e950c4faf8a4 WatchSource:0}: Error finding container ece1cfe70b85e25121862806093a5b04e6084c10157337f08612e950c4faf8a4: Status 404 returned error can't find the container with id ece1cfe70b85e25121862806093a5b04e6084c10157337f08612e950c4faf8a4 Sep 08 04:06:56.080905 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:56.080836 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58d7df9668-hgqgf" event={"ID":"c5b842e7-f15f-4453-bb7e-8191c1951874","Type":"ContainerStarted","Data":"ece1cfe70b85e25121862806093a5b04e6084c10157337f08612e950c4faf8a4"} Sep 08 04:06:56.173486 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:56.173455 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:56.281566 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:56.281532 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:56.666067 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:56.666031 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:57.058651 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:57.058616 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:58.188544 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:58.188492 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:58.250383 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:58.250353 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:06:59.282432 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:06:59.282404 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:00.099376 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.099342 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58d7df9668-hgqgf" event={"ID":"c5b842e7-f15f-4453-bb7e-8191c1951874","Type":"ContainerStarted","Data":"2d6b3de25e933a781d2e69cdadaa067ddf8293c6b85ebc716d5b6c4b4daf122d"} Sep 08 04:07:00.099542 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.099469 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-58d7df9668-hgqgf" Sep 08 04:07:00.115468 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.115424 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-58d7df9668-hgqgf" podStartSLOduration=1.8348888319999999 podStartE2EDuration="5.115412133s" totalImagesPullingTime="3.280523301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:06:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:07:00.114032483 +0000 UTC m=+394.036493263" watchObservedRunningTime="2026-09-08 04:07:00.115412133 +0000 UTC m=+394.037872873" Sep 08 04:07:00.327278 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.327252 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69cd8577c5-qwb8s"] Sep 08 04:07:00.330611 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.330594 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69cd8577c5-qwb8s" Sep 08 04:07:00.342388 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.342364 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69cd8577c5-qwb8s"] Sep 08 04:07:00.377755 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.377697 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/afa004f7-56b9-4b11-a01d-f14fb3491d06-console-config\") pod \"console-69cd8577c5-qwb8s\" (UID: \"afa004f7-56b9-4b11-a01d-f14fb3491d06\") " pod="openshift-console/console-69cd8577c5-qwb8s" Sep 08 04:07:00.377755 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.377729 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/afa004f7-56b9-4b11-a01d-f14fb3491d06-oauth-serving-cert\") pod \"console-69cd8577c5-qwb8s\" (UID: \"afa004f7-56b9-4b11-a01d-f14fb3491d06\") " pod="openshift-console/console-69cd8577c5-qwb8s" Sep 08 04:07:00.377878 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.377748 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4vkn\" (UniqueName: \"kubernetes.io/projected/afa004f7-56b9-4b11-a01d-f14fb3491d06-kube-api-access-l4vkn\") pod \"console-69cd8577c5-qwb8s\" (UID: \"afa004f7-56b9-4b11-a01d-f14fb3491d06\") " pod="openshift-console/console-69cd8577c5-qwb8s" Sep 08 04:07:00.377878 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.377826 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afa004f7-56b9-4b11-a01d-f14fb3491d06-trusted-ca-bundle\") pod \"console-69cd8577c5-qwb8s\" (UID: \"afa004f7-56b9-4b11-a01d-f14fb3491d06\") " pod="openshift-console/console-69cd8577c5-qwb8s" Sep 08 04:07:00.377990 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.377894 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/afa004f7-56b9-4b11-a01d-f14fb3491d06-console-oauth-config\") pod \"console-69cd8577c5-qwb8s\" (UID: \"afa004f7-56b9-4b11-a01d-f14fb3491d06\") " pod="openshift-console/console-69cd8577c5-qwb8s" Sep 08 04:07:00.377990 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.377925 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/afa004f7-56b9-4b11-a01d-f14fb3491d06-console-serving-cert\") pod \"console-69cd8577c5-qwb8s\" (UID: \"afa004f7-56b9-4b11-a01d-f14fb3491d06\") " pod="openshift-console/console-69cd8577c5-qwb8s" Sep 08 04:07:00.377990 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.377959 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/afa004f7-56b9-4b11-a01d-f14fb3491d06-service-ca\") pod \"console-69cd8577c5-qwb8s\" (UID: \"afa004f7-56b9-4b11-a01d-f14fb3491d06\") " pod="openshift-console/console-69cd8577c5-qwb8s" Sep 08 04:07:00.381523 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.381487 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:00.437896 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.437871 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:00.479175 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.479157 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afa004f7-56b9-4b11-a01d-f14fb3491d06-trusted-ca-bundle\") pod \"console-69cd8577c5-qwb8s\" (UID: \"afa004f7-56b9-4b11-a01d-f14fb3491d06\") " pod="openshift-console/console-69cd8577c5-qwb8s" Sep 08 04:07:00.479267 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.479198 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/afa004f7-56b9-4b11-a01d-f14fb3491d06-console-oauth-config\") pod \"console-69cd8577c5-qwb8s\" (UID: \"afa004f7-56b9-4b11-a01d-f14fb3491d06\") " pod="openshift-console/console-69cd8577c5-qwb8s" Sep 08 04:07:00.479267 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.479219 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/afa004f7-56b9-4b11-a01d-f14fb3491d06-console-serving-cert\") pod \"console-69cd8577c5-qwb8s\" (UID: \"afa004f7-56b9-4b11-a01d-f14fb3491d06\") " pod="openshift-console/console-69cd8577c5-qwb8s" Sep 08 04:07:00.479267 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.479240 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/afa004f7-56b9-4b11-a01d-f14fb3491d06-service-ca\") pod \"console-69cd8577c5-qwb8s\" (UID: \"afa004f7-56b9-4b11-a01d-f14fb3491d06\") " pod="openshift-console/console-69cd8577c5-qwb8s" Sep 08 04:07:00.479267 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.479260 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/afa004f7-56b9-4b11-a01d-f14fb3491d06-console-config\") pod \"console-69cd8577c5-qwb8s\" (UID: \"afa004f7-56b9-4b11-a01d-f14fb3491d06\") " pod="openshift-console/console-69cd8577c5-qwb8s" Sep 08 04:07:00.479461 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.479286 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/afa004f7-56b9-4b11-a01d-f14fb3491d06-oauth-serving-cert\") pod \"console-69cd8577c5-qwb8s\" (UID: \"afa004f7-56b9-4b11-a01d-f14fb3491d06\") " pod="openshift-console/console-69cd8577c5-qwb8s" Sep 08 04:07:00.479461 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.479312 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4vkn\" (UniqueName: \"kubernetes.io/projected/afa004f7-56b9-4b11-a01d-f14fb3491d06-kube-api-access-l4vkn\") pod \"console-69cd8577c5-qwb8s\" (UID: \"afa004f7-56b9-4b11-a01d-f14fb3491d06\") " pod="openshift-console/console-69cd8577c5-qwb8s" Sep 08 04:07:00.480010 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.479988 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/afa004f7-56b9-4b11-a01d-f14fb3491d06-service-ca\") pod \"console-69cd8577c5-qwb8s\" (UID: \"afa004f7-56b9-4b11-a01d-f14fb3491d06\") " pod="openshift-console/console-69cd8577c5-qwb8s" Sep 08 04:07:00.480090 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.480034 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/afa004f7-56b9-4b11-a01d-f14fb3491d06-oauth-serving-cert\") pod \"console-69cd8577c5-qwb8s\" (UID: \"afa004f7-56b9-4b11-a01d-f14fb3491d06\") " pod="openshift-console/console-69cd8577c5-qwb8s" Sep 08 04:07:00.480160 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.480137 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/afa004f7-56b9-4b11-a01d-f14fb3491d06-console-config\") pod \"console-69cd8577c5-qwb8s\" (UID: \"afa004f7-56b9-4b11-a01d-f14fb3491d06\") " pod="openshift-console/console-69cd8577c5-qwb8s" Sep 08 04:07:00.480343 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.480320 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afa004f7-56b9-4b11-a01d-f14fb3491d06-trusted-ca-bundle\") pod \"console-69cd8577c5-qwb8s\" (UID: \"afa004f7-56b9-4b11-a01d-f14fb3491d06\") " pod="openshift-console/console-69cd8577c5-qwb8s" Sep 08 04:07:00.481705 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.481682 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/afa004f7-56b9-4b11-a01d-f14fb3491d06-console-oauth-config\") pod \"console-69cd8577c5-qwb8s\" (UID: \"afa004f7-56b9-4b11-a01d-f14fb3491d06\") " pod="openshift-console/console-69cd8577c5-qwb8s" Sep 08 04:07:00.481834 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.481817 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/afa004f7-56b9-4b11-a01d-f14fb3491d06-console-serving-cert\") pod \"console-69cd8577c5-qwb8s\" (UID: \"afa004f7-56b9-4b11-a01d-f14fb3491d06\") " pod="openshift-console/console-69cd8577c5-qwb8s" Sep 08 04:07:00.487430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.487404 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4vkn\" (UniqueName: \"kubernetes.io/projected/afa004f7-56b9-4b11-a01d-f14fb3491d06-kube-api-access-l4vkn\") pod \"console-69cd8577c5-qwb8s\" (UID: \"afa004f7-56b9-4b11-a01d-f14fb3491d06\") " pod="openshift-console/console-69cd8577c5-qwb8s" Sep 08 04:07:00.535264 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.535245 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:00.637087 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.637036 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:00.639978 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.639961 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69cd8577c5-qwb8s" Sep 08 04:07:00.914109 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:00.914085 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69cd8577c5-qwb8s"] Sep 08 04:07:00.916053 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:07:00.916025 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafa004f7_56b9_4b11_a01d_f14fb3491d06.slice/crio-a47a56e06971f681ffce5944df2def2e37f4f80e15b222485df1ecbd3aff8157 WatchSource:0}: Error finding container a47a56e06971f681ffce5944df2def2e37f4f80e15b222485df1ecbd3aff8157: Status 404 returned error can't find the container with id a47a56e06971f681ffce5944df2def2e37f4f80e15b222485df1ecbd3aff8157 Sep 08 04:07:01.105658 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:01.105627 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69cd8577c5-qwb8s" event={"ID":"afa004f7-56b9-4b11-a01d-f14fb3491d06","Type":"ContainerStarted","Data":"f2fbdf759a19cbcfbdd2a60e19fc98408b88050e0a6314a56f5e2c6d88580e62"} Sep 08 04:07:01.105833 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:01.105678 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69cd8577c5-qwb8s" event={"ID":"afa004f7-56b9-4b11-a01d-f14fb3491d06","Type":"ContainerStarted","Data":"a47a56e06971f681ffce5944df2def2e37f4f80e15b222485df1ecbd3aff8157"} Sep 08 04:07:01.127283 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:01.127234 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69cd8577c5-qwb8s" podStartSLOduration=1.127221198 podStartE2EDuration="1.127221198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:07:01.125734226 +0000 UTC m=+395.048194989" watchObservedRunningTime="2026-09-08 04:07:01.127221198 +0000 UTC m=+395.049681937" Sep 08 04:07:01.130023 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:01.130000 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:01.230422 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:01.230365 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:02.805252 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:02.805221 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:02.905522 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:02.905482 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:03.468706 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.468674 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd"] Sep 08 04:07:03.471958 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.471938 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.475110 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.475085 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-pwzxr\"" Sep 08 04:07:03.475228 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.475111 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 08 04:07:03.485781 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.485762 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd"] Sep 08 04:07:03.501762 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.501739 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2678a7f5-2092-47d8-828f-1dd24f01b9e2-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd\" (UID: \"2678a7f5-2092-47d8-828f-1dd24f01b9e2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.501850 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.501769 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2678a7f5-2092-47d8-828f-1dd24f01b9e2-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd\" (UID: \"2678a7f5-2092-47d8-828f-1dd24f01b9e2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.501850 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.501786 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2678a7f5-2092-47d8-828f-1dd24f01b9e2-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd\" (UID: \"2678a7f5-2092-47d8-828f-1dd24f01b9e2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.501850 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.501812 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2678a7f5-2092-47d8-828f-1dd24f01b9e2-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd\" (UID: \"2678a7f5-2092-47d8-828f-1dd24f01b9e2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.501850 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.501846 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2678a7f5-2092-47d8-828f-1dd24f01b9e2-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd\" (UID: \"2678a7f5-2092-47d8-828f-1dd24f01b9e2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.502043 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.501925 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2678a7f5-2092-47d8-828f-1dd24f01b9e2-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd\" (UID: \"2678a7f5-2092-47d8-828f-1dd24f01b9e2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.502043 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.501948 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2678a7f5-2092-47d8-828f-1dd24f01b9e2-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd\" (UID: \"2678a7f5-2092-47d8-828f-1dd24f01b9e2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.502043 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.501972 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2678a7f5-2092-47d8-828f-1dd24f01b9e2-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd\" (UID: \"2678a7f5-2092-47d8-828f-1dd24f01b9e2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.502043 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.501998 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv48n\" (UniqueName: \"kubernetes.io/projected/2678a7f5-2092-47d8-828f-1dd24f01b9e2-kube-api-access-bv48n\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd\" (UID: \"2678a7f5-2092-47d8-828f-1dd24f01b9e2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.502741 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.502722 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:03.602476 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.602424 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2678a7f5-2092-47d8-828f-1dd24f01b9e2-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd\" (UID: \"2678a7f5-2092-47d8-828f-1dd24f01b9e2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.602476 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.602454 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2678a7f5-2092-47d8-828f-1dd24f01b9e2-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd\" (UID: \"2678a7f5-2092-47d8-828f-1dd24f01b9e2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.602476 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.602470 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2678a7f5-2092-47d8-828f-1dd24f01b9e2-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd\" (UID: \"2678a7f5-2092-47d8-828f-1dd24f01b9e2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.602647 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.602490 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv48n\" (UniqueName: \"kubernetes.io/projected/2678a7f5-2092-47d8-828f-1dd24f01b9e2-kube-api-access-bv48n\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd\" (UID: \"2678a7f5-2092-47d8-828f-1dd24f01b9e2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.602647 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.602575 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2678a7f5-2092-47d8-828f-1dd24f01b9e2-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd\" (UID: \"2678a7f5-2092-47d8-828f-1dd24f01b9e2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.602647 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.602604 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2678a7f5-2092-47d8-828f-1dd24f01b9e2-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd\" (UID: \"2678a7f5-2092-47d8-828f-1dd24f01b9e2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.602647 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.602628 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2678a7f5-2092-47d8-828f-1dd24f01b9e2-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd\" (UID: \"2678a7f5-2092-47d8-828f-1dd24f01b9e2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.602829 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.602659 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2678a7f5-2092-47d8-828f-1dd24f01b9e2-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd\" (UID: \"2678a7f5-2092-47d8-828f-1dd24f01b9e2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.602829 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.602803 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2678a7f5-2092-47d8-828f-1dd24f01b9e2-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd\" (UID: \"2678a7f5-2092-47d8-828f-1dd24f01b9e2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.602934 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.602876 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2678a7f5-2092-47d8-828f-1dd24f01b9e2-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd\" (UID: \"2678a7f5-2092-47d8-828f-1dd24f01b9e2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.603021 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.602996 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2678a7f5-2092-47d8-828f-1dd24f01b9e2-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd\" (UID: \"2678a7f5-2092-47d8-828f-1dd24f01b9e2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.603079 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.603029 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2678a7f5-2092-47d8-828f-1dd24f01b9e2-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd\" (UID: \"2678a7f5-2092-47d8-828f-1dd24f01b9e2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.603184 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.603166 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2678a7f5-2092-47d8-828f-1dd24f01b9e2-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd\" (UID: \"2678a7f5-2092-47d8-828f-1dd24f01b9e2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.603242 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.603171 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2678a7f5-2092-47d8-828f-1dd24f01b9e2-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd\" (UID: \"2678a7f5-2092-47d8-828f-1dd24f01b9e2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.604916 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.604894 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2678a7f5-2092-47d8-828f-1dd24f01b9e2-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd\" (UID: \"2678a7f5-2092-47d8-828f-1dd24f01b9e2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.605147 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.605122 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2678a7f5-2092-47d8-828f-1dd24f01b9e2-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd\" (UID: \"2678a7f5-2092-47d8-828f-1dd24f01b9e2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.610368 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.610349 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2678a7f5-2092-47d8-828f-1dd24f01b9e2-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd\" (UID: \"2678a7f5-2092-47d8-828f-1dd24f01b9e2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.610568 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.610546 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv48n\" (UniqueName: \"kubernetes.io/projected/2678a7f5-2092-47d8-828f-1dd24f01b9e2-kube-api-access-bv48n\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd\" (UID: \"2678a7f5-2092-47d8-828f-1dd24f01b9e2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:03.715409 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.715383 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:03.783249 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:03.783226 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:04.065247 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:04.065225 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd"] Sep 08 04:07:04.067157 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:07:04.067130 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2678a7f5_2092_47d8_828f_1dd24f01b9e2.slice/crio-9b927ad0df62bb9e9246fc8341873643bf2d319612fb20cfe1548c40a83c798c WatchSource:0}: Error finding container 9b927ad0df62bb9e9246fc8341873643bf2d319612fb20cfe1548c40a83c798c: Status 404 returned error can't find the container with id 9b927ad0df62bb9e9246fc8341873643bf2d319612fb20cfe1548c40a83c798c Sep 08 04:07:04.118372 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:04.118345 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" event={"ID":"2678a7f5-2092-47d8-828f-1dd24f01b9e2","Type":"ContainerStarted","Data":"9b927ad0df62bb9e9246fc8341873643bf2d319612fb20cfe1548c40a83c798c"} Sep 08 04:07:04.151493 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:04.151466 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:06.720941 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:06.720893 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:07:06.721234 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:06.720965 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:07:06.721234 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:06.720997 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:07:07.137179 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:07.137127 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" event={"ID":"2678a7f5-2092-47d8-828f-1dd24f01b9e2","Type":"ContainerStarted","Data":"9390311994c600ea86e8a62bc9fe569399d5a6018f31e2789971a9984eef5451"} Sep 08 04:07:07.159403 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:07.159352 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" podStartSLOduration=1.5081630590000001 podStartE2EDuration="4.159337516s" totalImagesPullingTime="2.651174457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:07:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:07:07.158108598 +0000 UTC m=+401.080569356" watchObservedRunningTime="2026-09-08 04:07:07.159337516 +0000 UTC m=+401.081798256" Sep 08 04:07:07.381057 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:07.381022 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-nwlcc"] Sep 08 04:07:07.385672 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:07.385654 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-nwlcc" Sep 08 04:07:07.388259 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:07.388206 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-4ppkk\"" Sep 08 04:07:07.388259 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:07.388244 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 08 04:07:07.392183 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:07.392144 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-nwlcc"] Sep 08 04:07:07.405944 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:07.405917 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:07.433653 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:07.433622 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f45744aa-dccc-43d2-be9a-dc182ba87805-cert\") pod \"odh-model-controller-7998c67dc4-nwlcc\" (UID: \"f45744aa-dccc-43d2-be9a-dc182ba87805\") " pod="opendatahub/odh-model-controller-7998c67dc4-nwlcc" Sep 08 04:07:07.433793 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:07.433696 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x27xx\" (UniqueName: \"kubernetes.io/projected/f45744aa-dccc-43d2-be9a-dc182ba87805-kube-api-access-x27xx\") pod \"odh-model-controller-7998c67dc4-nwlcc\" (UID: \"f45744aa-dccc-43d2-be9a-dc182ba87805\") " pod="opendatahub/odh-model-controller-7998c67dc4-nwlcc" Sep 08 04:07:07.465796 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:07.465747 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:07.535146 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:07.535117 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x27xx\" (UniqueName: \"kubernetes.io/projected/f45744aa-dccc-43d2-be9a-dc182ba87805-kube-api-access-x27xx\") pod \"odh-model-controller-7998c67dc4-nwlcc\" (UID: \"f45744aa-dccc-43d2-be9a-dc182ba87805\") " pod="opendatahub/odh-model-controller-7998c67dc4-nwlcc" Sep 08 04:07:07.535307 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:07.535175 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f45744aa-dccc-43d2-be9a-dc182ba87805-cert\") pod \"odh-model-controller-7998c67dc4-nwlcc\" (UID: \"f45744aa-dccc-43d2-be9a-dc182ba87805\") " pod="opendatahub/odh-model-controller-7998c67dc4-nwlcc" Sep 08 04:07:07.535307 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:07:07.535277 2574 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 08 04:07:07.535418 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:07:07.535338 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f45744aa-dccc-43d2-be9a-dc182ba87805-cert podName:f45744aa-dccc-43d2-be9a-dc182ba87805 nodeName:}" failed. No retries permitted until 2026-09-08 04:07:08.03531867 +0000 UTC m=+401.957779394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f45744aa-dccc-43d2-be9a-dc182ba87805-cert") pod "odh-model-controller-7998c67dc4-nwlcc" (UID: "f45744aa-dccc-43d2-be9a-dc182ba87805") : secret "odh-model-controller-webhook-cert" not found Sep 08 04:07:07.547297 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:07.547266 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x27xx\" (UniqueName: \"kubernetes.io/projected/f45744aa-dccc-43d2-be9a-dc182ba87805-kube-api-access-x27xx\") pod \"odh-model-controller-7998c67dc4-nwlcc\" (UID: \"f45744aa-dccc-43d2-be9a-dc182ba87805\") " pod="opendatahub/odh-model-controller-7998c67dc4-nwlcc" Sep 08 04:07:07.565186 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:07.565154 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:07.665767 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:07.665682 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:07.784351 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:07.784313 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:07.788952 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:07.788927 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:07.792715 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:07.792686 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:07.955150 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:07.955075 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:07.976638 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:07.976607 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vxz9r"] Sep 08 04:07:07.979920 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:07.979902 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vxz9r" Sep 08 04:07:07.983111 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:07.983093 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 08 04:07:07.983325 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:07.983313 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-spwc6\"" Sep 08 04:07:07.991637 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:07.991614 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vxz9r"] Sep 08 04:07:08.040516 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:08.040473 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db0e3e2-e65f-40fb-9150-fd7d087e5de9-tls-certs\") pod \"model-serving-api-54d89bd5fc-vxz9r\" (UID: \"6db0e3e2-e65f-40fb-9150-fd7d087e5de9\") " pod="opendatahub/model-serving-api-54d89bd5fc-vxz9r" Sep 08 04:07:08.040665 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:08.040590 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f45744aa-dccc-43d2-be9a-dc182ba87805-cert\") pod \"odh-model-controller-7998c67dc4-nwlcc\" (UID: \"f45744aa-dccc-43d2-be9a-dc182ba87805\") " pod="opendatahub/odh-model-controller-7998c67dc4-nwlcc" Sep 08 04:07:08.040665 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:08.040611 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f77hb\" (UniqueName: \"kubernetes.io/projected/6db0e3e2-e65f-40fb-9150-fd7d087e5de9-kube-api-access-f77hb\") pod \"model-serving-api-54d89bd5fc-vxz9r\" (UID: \"6db0e3e2-e65f-40fb-9150-fd7d087e5de9\") " pod="opendatahub/model-serving-api-54d89bd5fc-vxz9r" Sep 08 04:07:08.043155 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:08.043133 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f45744aa-dccc-43d2-be9a-dc182ba87805-cert\") pod \"odh-model-controller-7998c67dc4-nwlcc\" (UID: \"f45744aa-dccc-43d2-be9a-dc182ba87805\") " pod="opendatahub/odh-model-controller-7998c67dc4-nwlcc" Sep 08 04:07:08.141205 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:08.141172 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db0e3e2-e65f-40fb-9150-fd7d087e5de9-tls-certs\") pod \"model-serving-api-54d89bd5fc-vxz9r\" (UID: \"6db0e3e2-e65f-40fb-9150-fd7d087e5de9\") " pod="opendatahub/model-serving-api-54d89bd5fc-vxz9r" Sep 08 04:07:08.141345 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:08.141261 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f77hb\" (UniqueName: \"kubernetes.io/projected/6db0e3e2-e65f-40fb-9150-fd7d087e5de9-kube-api-access-f77hb\") pod \"model-serving-api-54d89bd5fc-vxz9r\" (UID: \"6db0e3e2-e65f-40fb-9150-fd7d087e5de9\") " pod="opendatahub/model-serving-api-54d89bd5fc-vxz9r" Sep 08 04:07:08.141402 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:07:08.141337 2574 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 08 04:07:08.141438 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:07:08.141418 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6db0e3e2-e65f-40fb-9150-fd7d087e5de9-tls-certs podName:6db0e3e2-e65f-40fb-9150-fd7d087e5de9 nodeName:}" failed. No retries permitted until 2026-09-08 04:07:08.641397178 +0000 UTC m=+402.563857917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6db0e3e2-e65f-40fb-9150-fd7d087e5de9-tls-certs") pod "model-serving-api-54d89bd5fc-vxz9r" (UID: "6db0e3e2-e65f-40fb-9150-fd7d087e5de9") : secret "model-serving-api-tls" not found Sep 08 04:07:08.142326 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:08.142303 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:08.143412 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:08.143393 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd" Sep 08 04:07:08.149993 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:08.149968 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f77hb\" (UniqueName: \"kubernetes.io/projected/6db0e3e2-e65f-40fb-9150-fd7d087e5de9-kube-api-access-f77hb\") pod \"model-serving-api-54d89bd5fc-vxz9r\" (UID: \"6db0e3e2-e65f-40fb-9150-fd7d087e5de9\") " pod="opendatahub/model-serving-api-54d89bd5fc-vxz9r" Sep 08 04:07:08.197788 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:08.197762 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:08.299782 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:08.299746 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-nwlcc" Sep 08 04:07:08.575396 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:08.575310 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-nwlcc"] Sep 08 04:07:08.577923 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:07:08.577894 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45744aa_dccc_43d2_be9a_dc182ba87805.slice/crio-822580dd2bfbc409bec893a20d04011a585efc99e4c888b5928c84b936daa123 WatchSource:0}: Error finding container 822580dd2bfbc409bec893a20d04011a585efc99e4c888b5928c84b936daa123: Status 404 returned error can't find the container with id 822580dd2bfbc409bec893a20d04011a585efc99e4c888b5928c84b936daa123 Sep 08 04:07:08.646182 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:08.646153 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db0e3e2-e65f-40fb-9150-fd7d087e5de9-tls-certs\") pod \"model-serving-api-54d89bd5fc-vxz9r\" (UID: \"6db0e3e2-e65f-40fb-9150-fd7d087e5de9\") " pod="opendatahub/model-serving-api-54d89bd5fc-vxz9r" Sep 08 04:07:08.648524 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:08.648484 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db0e3e2-e65f-40fb-9150-fd7d087e5de9-tls-certs\") pod \"model-serving-api-54d89bd5fc-vxz9r\" (UID: \"6db0e3e2-e65f-40fb-9150-fd7d087e5de9\") " pod="opendatahub/model-serving-api-54d89bd5fc-vxz9r" Sep 08 04:07:08.889985 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:08.889908 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vxz9r" Sep 08 04:07:09.148726 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:09.148557 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-nwlcc" event={"ID":"f45744aa-dccc-43d2-be9a-dc182ba87805","Type":"ContainerStarted","Data":"822580dd2bfbc409bec893a20d04011a585efc99e4c888b5928c84b936daa123"} Sep 08 04:07:09.170230 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:09.170171 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vxz9r"] Sep 08 04:07:09.173279 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:07:09.173247 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db0e3e2_e65f_40fb_9150_fd7d087e5de9.slice/crio-46f68266cb27bd51496a2f24457f211c1b2f220590fb75da39a37a89e643ef5f WatchSource:0}: Error finding container 46f68266cb27bd51496a2f24457f211c1b2f220590fb75da39a37a89e643ef5f: Status 404 returned error can't find the container with id 46f68266cb27bd51496a2f24457f211c1b2f220590fb75da39a37a89e643ef5f Sep 08 04:07:10.157350 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:10.157287 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vxz9r" event={"ID":"6db0e3e2-e65f-40fb-9150-fd7d087e5de9","Type":"ContainerStarted","Data":"46f68266cb27bd51496a2f24457f211c1b2f220590fb75da39a37a89e643ef5f"} Sep 08 04:07:10.640913 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:10.640862 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69cd8577c5-qwb8s" Sep 08 04:07:10.640913 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:10.640915 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69cd8577c5-qwb8s" Sep 08 04:07:10.647753 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:10.647450 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69cd8577c5-qwb8s" Sep 08 04:07:11.107975 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:11.107942 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-58d7df9668-hgqgf" Sep 08 04:07:11.166941 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:11.166909 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69cd8577c5-qwb8s" Sep 08 04:07:11.214937 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:11.214866 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75dd5cddb5-n9dc7"] Sep 08 04:07:11.321042 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:11.321005 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:11.421677 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:11.421602 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:11.479496 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:11.479468 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:13.172743 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:13.172706 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vxz9r" event={"ID":"6db0e3e2-e65f-40fb-9150-fd7d087e5de9","Type":"ContainerStarted","Data":"222a7a4c1d0d4b746ceb965620054eee11f8631ba4998a59579fa030c4dc2c04"} Sep 08 04:07:13.173192 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:13.172766 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-vxz9r" Sep 08 04:07:13.174197 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:13.174171 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-nwlcc" event={"ID":"f45744aa-dccc-43d2-be9a-dc182ba87805","Type":"ContainerStarted","Data":"86fa54186f4b36f8bb661c2d14bf64373f0a3429873f274aa63ff6e8fbbe718e"} Sep 08 04:07:13.174334 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:13.174319 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-nwlcc" Sep 08 04:07:13.192380 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:13.192323 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-vxz9r" podStartSLOduration=2.584348101 podStartE2EDuration="6.192304547s" totalImagesPullingTime="3.607956446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:07:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:07:13.191167439 +0000 UTC m=+407.113628203" watchObservedRunningTime="2026-09-08 04:07:13.192304547 +0000 UTC m=+407.114765288" Sep 08 04:07:13.207392 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:13.207344 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-nwlcc" podStartSLOduration=2.039013508 podStartE2EDuration="6.20732911s" totalImagesPullingTime="4.168315602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:07:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:07:13.206699354 +0000 UTC m=+407.129160119" watchObservedRunningTime="2026-09-08 04:07:13.20732911 +0000 UTC m=+407.129789873" Sep 08 04:07:14.720674 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.720642 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg"] Sep 08 04:07:14.723972 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.723957 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" Sep 08 04:07:14.727262 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.727237 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-n5zmn\"" Sep 08 04:07:14.727382 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.727327 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 08 04:07:14.727382 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.727327 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 08 04:07:14.727533 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.727518 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 08 04:07:14.745861 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.745837 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg"] Sep 08 04:07:14.802178 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.802148 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f4a49d-e2d6-4f39-9648-c55322d28666-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-p94kg\" (UID: \"b0f4a49d-e2d6-4f39-9648-c55322d28666\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" Sep 08 04:07:14.802284 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.802197 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b0f4a49d-e2d6-4f39-9648-c55322d28666-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-p94kg\" (UID: \"b0f4a49d-e2d6-4f39-9648-c55322d28666\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" Sep 08 04:07:14.802284 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.802271 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0f4a49d-e2d6-4f39-9648-c55322d28666-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-p94kg\" (UID: \"b0f4a49d-e2d6-4f39-9648-c55322d28666\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" Sep 08 04:07:14.802366 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.802314 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b0f4a49d-e2d6-4f39-9648-c55322d28666-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-p94kg\" (UID: \"b0f4a49d-e2d6-4f39-9648-c55322d28666\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" Sep 08 04:07:14.802366 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.802344 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b0f4a49d-e2d6-4f39-9648-c55322d28666-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-p94kg\" (UID: \"b0f4a49d-e2d6-4f39-9648-c55322d28666\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" Sep 08 04:07:14.802451 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.802377 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b0f4a49d-e2d6-4f39-9648-c55322d28666-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-p94kg\" (UID: \"b0f4a49d-e2d6-4f39-9648-c55322d28666\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" Sep 08 04:07:14.802451 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.802408 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxztj\" (UniqueName: \"kubernetes.io/projected/b0f4a49d-e2d6-4f39-9648-c55322d28666-kube-api-access-xxztj\") pod \"istiod-openshift-gateway-5677ccb49b-p94kg\" (UID: \"b0f4a49d-e2d6-4f39-9648-c55322d28666\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" Sep 08 04:07:14.902908 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.902876 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b0f4a49d-e2d6-4f39-9648-c55322d28666-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-p94kg\" (UID: \"b0f4a49d-e2d6-4f39-9648-c55322d28666\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" Sep 08 04:07:14.903020 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.902916 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxztj\" (UniqueName: \"kubernetes.io/projected/b0f4a49d-e2d6-4f39-9648-c55322d28666-kube-api-access-xxztj\") pod \"istiod-openshift-gateway-5677ccb49b-p94kg\" (UID: \"b0f4a49d-e2d6-4f39-9648-c55322d28666\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" Sep 08 04:07:14.903020 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.902949 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f4a49d-e2d6-4f39-9648-c55322d28666-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-p94kg\" (UID: \"b0f4a49d-e2d6-4f39-9648-c55322d28666\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" Sep 08 04:07:14.903020 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.902973 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b0f4a49d-e2d6-4f39-9648-c55322d28666-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-p94kg\" (UID: \"b0f4a49d-e2d6-4f39-9648-c55322d28666\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" Sep 08 04:07:14.903020 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.903013 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0f4a49d-e2d6-4f39-9648-c55322d28666-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-p94kg\" (UID: \"b0f4a49d-e2d6-4f39-9648-c55322d28666\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" Sep 08 04:07:14.903237 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.903043 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b0f4a49d-e2d6-4f39-9648-c55322d28666-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-p94kg\" (UID: \"b0f4a49d-e2d6-4f39-9648-c55322d28666\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" Sep 08 04:07:14.903237 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.903097 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b0f4a49d-e2d6-4f39-9648-c55322d28666-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-p94kg\" (UID: \"b0f4a49d-e2d6-4f39-9648-c55322d28666\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" Sep 08 04:07:14.903663 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.903643 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b0f4a49d-e2d6-4f39-9648-c55322d28666-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-p94kg\" (UID: \"b0f4a49d-e2d6-4f39-9648-c55322d28666\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" Sep 08 04:07:14.905262 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.905245 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0f4a49d-e2d6-4f39-9648-c55322d28666-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-p94kg\" (UID: \"b0f4a49d-e2d6-4f39-9648-c55322d28666\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" Sep 08 04:07:14.905518 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.905482 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b0f4a49d-e2d6-4f39-9648-c55322d28666-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-p94kg\" (UID: \"b0f4a49d-e2d6-4f39-9648-c55322d28666\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" Sep 08 04:07:14.905700 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.905682 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b0f4a49d-e2d6-4f39-9648-c55322d28666-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-p94kg\" (UID: \"b0f4a49d-e2d6-4f39-9648-c55322d28666\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" Sep 08 04:07:14.905763 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.905686 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b0f4a49d-e2d6-4f39-9648-c55322d28666-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-p94kg\" (UID: \"b0f4a49d-e2d6-4f39-9648-c55322d28666\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" Sep 08 04:07:14.916985 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.916956 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxztj\" (UniqueName: \"kubernetes.io/projected/b0f4a49d-e2d6-4f39-9648-c55322d28666-kube-api-access-xxztj\") pod \"istiod-openshift-gateway-5677ccb49b-p94kg\" (UID: \"b0f4a49d-e2d6-4f39-9648-c55322d28666\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" Sep 08 04:07:14.920805 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:14.920786 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0f4a49d-e2d6-4f39-9648-c55322d28666-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-p94kg\" (UID: \"b0f4a49d-e2d6-4f39-9648-c55322d28666\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" Sep 08 04:07:15.032717 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:15.032695 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" Sep 08 04:07:15.332217 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:15.332185 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg"] Sep 08 04:07:15.333912 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:07:15.333882 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f4a49d_e2d6_4f39_9648_c55322d28666.slice/crio-abf52b952f86bfbb7abb990a423a9913026bf925b0e1887c708c265d41c1e5d3 WatchSource:0}: Error finding container abf52b952f86bfbb7abb990a423a9913026bf925b0e1887c708c265d41c1e5d3: Status 404 returned error can't find the container with id abf52b952f86bfbb7abb990a423a9913026bf925b0e1887c708c265d41c1e5d3 Sep 08 04:07:16.189530 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:16.189480 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" event={"ID":"b0f4a49d-e2d6-4f39-9648-c55322d28666","Type":"ContainerStarted","Data":"abf52b952f86bfbb7abb990a423a9913026bf925b0e1887c708c265d41c1e5d3"} Sep 08 04:07:17.687872 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:17.687832 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:17.830391 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:17.830347 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:17.915962 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:17.915927 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:07:17.916065 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:17.916001 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:07:18.181299 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:18.181224 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:18.204868 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:18.204786 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" event={"ID":"b0f4a49d-e2d6-4f39-9648-c55322d28666","Type":"ContainerStarted","Data":"b97eeb9c28a10d364aa05b51486d7540ff1e5b2ccef789bfc786e09452559814"} Sep 08 04:07:18.205025 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:18.204924 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" Sep 08 04:07:18.206976 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:18.206949 2574 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-p94kg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 04:07:18.207092 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:18.206999 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" podUID="b0f4a49d-e2d6-4f39-9648-c55322d28666" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 04:07:18.250892 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:18.250827 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" podStartSLOduration=1.671215973 podStartE2EDuration="4.250805954s" totalImagesPullingTime="2.579589981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:07:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:07:18.250259487 +0000 UTC m=+412.172720242" watchObservedRunningTime="2026-09-08 04:07:18.250805954 +0000 UTC m=+412.173266696" Sep 08 04:07:18.304892 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:18.304858 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:19.022794 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:19.022759 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:19.139012 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:19.138983 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:19.209902 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:19.209873 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p94kg" Sep 08 04:07:19.283971 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:19.283908 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:24.181566 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:24.181539 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-nwlcc" Sep 08 04:07:24.183462 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:24.183443 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-vxz9r" Sep 08 04:07:33.179856 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:33.179824 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:35.022375 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:35.022345 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:35.079925 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:35.079902 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:35.550670 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:35.550632 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:36.240565 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:36.240479 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75dd5cddb5-n9dc7" podUID="d42e70e1-3982-4d6d-b254-564e116de345" containerName="console" containerID="cri-o://825612cb1f2a0c38292c9ec052d9e091e0dc894e03c4194a305bf00948a1b995" gracePeriod=15 Sep 08 04:07:36.484195 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:36.484172 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:07:36.570558 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:36.570533 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz8l7\" (UniqueName: \"kubernetes.io/projected/d42e70e1-3982-4d6d-b254-564e116de345-kube-api-access-lz8l7\") pod \"d42e70e1-3982-4d6d-b254-564e116de345\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " Sep 08 04:07:36.570704 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:36.570584 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d42e70e1-3982-4d6d-b254-564e116de345-console-oauth-config\") pod \"d42e70e1-3982-4d6d-b254-564e116de345\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " Sep 08 04:07:36.570704 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:36.570615 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d42e70e1-3982-4d6d-b254-564e116de345-oauth-serving-cert\") pod \"d42e70e1-3982-4d6d-b254-564e116de345\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " Sep 08 04:07:36.570704 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:36.570643 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d42e70e1-3982-4d6d-b254-564e116de345-console-config\") pod \"d42e70e1-3982-4d6d-b254-564e116de345\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " Sep 08 04:07:36.570704 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:36.570668 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d42e70e1-3982-4d6d-b254-564e116de345-trusted-ca-bundle\") pod \"d42e70e1-3982-4d6d-b254-564e116de345\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " Sep 08 04:07:36.570704 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:36.570695 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d42e70e1-3982-4d6d-b254-564e116de345-console-serving-cert\") pod \"d42e70e1-3982-4d6d-b254-564e116de345\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " Sep 08 04:07:36.570943 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:36.570717 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d42e70e1-3982-4d6d-b254-564e116de345-service-ca\") pod \"d42e70e1-3982-4d6d-b254-564e116de345\" (UID: \"d42e70e1-3982-4d6d-b254-564e116de345\") " Sep 08 04:07:36.571054 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:36.571010 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d42e70e1-3982-4d6d-b254-564e116de345-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d42e70e1-3982-4d6d-b254-564e116de345" (UID: "d42e70e1-3982-4d6d-b254-564e116de345"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:07:36.571301 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:36.571276 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d42e70e1-3982-4d6d-b254-564e116de345-service-ca" (OuterVolumeSpecName: "service-ca") pod "d42e70e1-3982-4d6d-b254-564e116de345" (UID: "d42e70e1-3982-4d6d-b254-564e116de345"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:07:36.571373 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:36.571306 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d42e70e1-3982-4d6d-b254-564e116de345-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d42e70e1-3982-4d6d-b254-564e116de345" (UID: "d42e70e1-3982-4d6d-b254-564e116de345"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:07:36.571373 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:36.571343 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d42e70e1-3982-4d6d-b254-564e116de345-console-config" (OuterVolumeSpecName: "console-config") pod "d42e70e1-3982-4d6d-b254-564e116de345" (UID: "d42e70e1-3982-4d6d-b254-564e116de345"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:07:36.572926 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:36.572902 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d42e70e1-3982-4d6d-b254-564e116de345-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d42e70e1-3982-4d6d-b254-564e116de345" (UID: "d42e70e1-3982-4d6d-b254-564e116de345"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 04:07:36.573093 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:36.573071 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d42e70e1-3982-4d6d-b254-564e116de345-kube-api-access-lz8l7" (OuterVolumeSpecName: "kube-api-access-lz8l7") pod "d42e70e1-3982-4d6d-b254-564e116de345" (UID: "d42e70e1-3982-4d6d-b254-564e116de345"). InnerVolumeSpecName "kube-api-access-lz8l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:07:36.573214 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:36.573193 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d42e70e1-3982-4d6d-b254-564e116de345-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d42e70e1-3982-4d6d-b254-564e116de345" (UID: "d42e70e1-3982-4d6d-b254-564e116de345"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 04:07:36.671693 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:36.671671 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz8l7\" (UniqueName: \"kubernetes.io/projected/d42e70e1-3982-4d6d-b254-564e116de345-kube-api-access-lz8l7\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:07:36.671693 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:36.671691 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d42e70e1-3982-4d6d-b254-564e116de345-console-oauth-config\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:07:36.671813 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:36.671700 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d42e70e1-3982-4d6d-b254-564e116de345-oauth-serving-cert\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:07:36.671813 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:36.671709 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d42e70e1-3982-4d6d-b254-564e116de345-console-config\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:07:36.671813 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:36.671717 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d42e70e1-3982-4d6d-b254-564e116de345-trusted-ca-bundle\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:07:36.671813 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:36.671727 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d42e70e1-3982-4d6d-b254-564e116de345-console-serving-cert\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:07:36.671813 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:36.671736 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d42e70e1-3982-4d6d-b254-564e116de345-service-ca\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:07:37.294714 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:37.294692 2574 generic.go:356] "Generic (PLEG): container finished" podID="d42e70e1-3982-4d6d-b254-564e116de345" containerID="825612cb1f2a0c38292c9ec052d9e091e0dc894e03c4194a305bf00948a1b995" exitCode=0 Sep 08 04:07:37.295082 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:37.294738 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75dd5cddb5-n9dc7" Sep 08 04:07:37.295082 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:37.294773 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75dd5cddb5-n9dc7" event={"ID":"d42e70e1-3982-4d6d-b254-564e116de345","Type":"ContainerDied","Data":"825612cb1f2a0c38292c9ec052d9e091e0dc894e03c4194a305bf00948a1b995"} Sep 08 04:07:37.295082 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:37.294831 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75dd5cddb5-n9dc7" event={"ID":"d42e70e1-3982-4d6d-b254-564e116de345","Type":"ContainerDied","Data":"93734bd53bbb526dccc7a907734ac5f438d82d39b49111b11e77710a21ea4a53"} Sep 08 04:07:37.295082 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:37.294852 2574 scope.go:117] "RemoveContainer" containerID="825612cb1f2a0c38292c9ec052d9e091e0dc894e03c4194a305bf00948a1b995" Sep 08 04:07:37.306691 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:37.306676 2574 scope.go:117] "RemoveContainer" containerID="825612cb1f2a0c38292c9ec052d9e091e0dc894e03c4194a305bf00948a1b995" Sep 08 04:07:37.306961 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:07:37.306943 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"825612cb1f2a0c38292c9ec052d9e091e0dc894e03c4194a305bf00948a1b995\": container with ID starting with 825612cb1f2a0c38292c9ec052d9e091e0dc894e03c4194a305bf00948a1b995 not found: ID does not exist" containerID="825612cb1f2a0c38292c9ec052d9e091e0dc894e03c4194a305bf00948a1b995" Sep 08 04:07:37.307009 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:37.306971 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"825612cb1f2a0c38292c9ec052d9e091e0dc894e03c4194a305bf00948a1b995"} err="failed to get container status \"825612cb1f2a0c38292c9ec052d9e091e0dc894e03c4194a305bf00948a1b995\": rpc error: code = NotFound desc = could not find container \"825612cb1f2a0c38292c9ec052d9e091e0dc894e03c4194a305bf00948a1b995\": container with ID starting with 825612cb1f2a0c38292c9ec052d9e091e0dc894e03c4194a305bf00948a1b995 not found: ID does not exist" Sep 08 04:07:37.317579 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:37.317560 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75dd5cddb5-n9dc7"] Sep 08 04:07:37.321134 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:37.321116 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75dd5cddb5-n9dc7"] Sep 08 04:07:38.566557 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:38.566524 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d42e70e1-3982-4d6d-b254-564e116de345" path="/var/lib/kubelet/pods/d42e70e1-3982-4d6d-b254-564e116de345/volumes" Sep 08 04:07:40.114463 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:40.114421 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:07:40.172839 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:07:40.172813 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:08.433789 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:08.433753 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:08.526076 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:08.526049 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:12.275605 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.275571 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g"] Sep 08 04:08:12.275955 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.275888 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d42e70e1-3982-4d6d-b254-564e116de345" containerName="console" Sep 08 04:08:12.275955 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.275899 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d42e70e1-3982-4d6d-b254-564e116de345" containerName="console" Sep 08 04:08:12.275955 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.275956 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d42e70e1-3982-4d6d-b254-564e116de345" containerName="console" Sep 08 04:08:12.282610 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.282588 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g" Sep 08 04:08:12.286922 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.286898 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 04:08:12.287222 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.286902 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-d97f6\"" Sep 08 04:08:12.287342 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.286917 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 04:08:12.290208 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.289909 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g"] Sep 08 04:08:12.366444 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.366421 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld"] Sep 08 04:08:12.370042 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.370026 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld" Sep 08 04:08:12.378195 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.378172 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld"] Sep 08 04:08:12.417069 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.417044 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07fe3c21-bee0-41a0-b6fb-31eaa110a2c1-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g\" (UID: \"07fe3c21-bee0-41a0-b6fb-31eaa110a2c1\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g" Sep 08 04:08:12.417166 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.417078 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvmcq\" (UniqueName: \"kubernetes.io/projected/07fe3c21-bee0-41a0-b6fb-31eaa110a2c1-kube-api-access-lvmcq\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g\" (UID: \"07fe3c21-bee0-41a0-b6fb-31eaa110a2c1\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g" Sep 08 04:08:12.417166 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.417111 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07fe3c21-bee0-41a0-b6fb-31eaa110a2c1-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g\" (UID: \"07fe3c21-bee0-41a0-b6fb-31eaa110a2c1\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g" Sep 08 04:08:12.471088 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.471062 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v"] Sep 08 04:08:12.474326 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.474311 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v" Sep 08 04:08:12.484550 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.484529 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v"] Sep 08 04:08:12.517625 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.517604 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e49228e3-a4ec-47b8-9ffd-7fc853c4639e-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld\" (UID: \"e49228e3-a4ec-47b8-9ffd-7fc853c4639e\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld" Sep 08 04:08:12.517736 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.517637 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h78wf\" (UniqueName: \"kubernetes.io/projected/e49228e3-a4ec-47b8-9ffd-7fc853c4639e-kube-api-access-h78wf\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld\" (UID: \"e49228e3-a4ec-47b8-9ffd-7fc853c4639e\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld" Sep 08 04:08:12.517736 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.517659 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07fe3c21-bee0-41a0-b6fb-31eaa110a2c1-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g\" (UID: \"07fe3c21-bee0-41a0-b6fb-31eaa110a2c1\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g" Sep 08 04:08:12.517736 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.517680 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvmcq\" (UniqueName: \"kubernetes.io/projected/07fe3c21-bee0-41a0-b6fb-31eaa110a2c1-kube-api-access-lvmcq\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g\" (UID: \"07fe3c21-bee0-41a0-b6fb-31eaa110a2c1\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g" Sep 08 04:08:12.517888 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.517756 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e49228e3-a4ec-47b8-9ffd-7fc853c4639e-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld\" (UID: \"e49228e3-a4ec-47b8-9ffd-7fc853c4639e\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld" Sep 08 04:08:12.517940 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.517888 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07fe3c21-bee0-41a0-b6fb-31eaa110a2c1-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g\" (UID: \"07fe3c21-bee0-41a0-b6fb-31eaa110a2c1\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g" Sep 08 04:08:12.518048 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.518029 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07fe3c21-bee0-41a0-b6fb-31eaa110a2c1-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g\" (UID: \"07fe3c21-bee0-41a0-b6fb-31eaa110a2c1\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g" Sep 08 04:08:12.518195 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.518177 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07fe3c21-bee0-41a0-b6fb-31eaa110a2c1-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g\" (UID: \"07fe3c21-bee0-41a0-b6fb-31eaa110a2c1\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g" Sep 08 04:08:12.526530 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.526462 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvmcq\" (UniqueName: \"kubernetes.io/projected/07fe3c21-bee0-41a0-b6fb-31eaa110a2c1-kube-api-access-lvmcq\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g\" (UID: \"07fe3c21-bee0-41a0-b6fb-31eaa110a2c1\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g" Sep 08 04:08:12.572486 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.572464 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr"] Sep 08 04:08:12.575985 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.575969 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr" Sep 08 04:08:12.584965 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.584945 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr"] Sep 08 04:08:12.592585 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.592569 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g" Sep 08 04:08:12.618278 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.618255 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22674c94-7b80-4ba9-8f07-5a890003c00a-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v\" (UID: \"22674c94-7b80-4ba9-8f07-5a890003c00a\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v" Sep 08 04:08:12.618371 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.618293 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e49228e3-a4ec-47b8-9ffd-7fc853c4639e-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld\" (UID: \"e49228e3-a4ec-47b8-9ffd-7fc853c4639e\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld" Sep 08 04:08:12.618371 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.618318 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h78wf\" (UniqueName: \"kubernetes.io/projected/e49228e3-a4ec-47b8-9ffd-7fc853c4639e-kube-api-access-h78wf\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld\" (UID: \"e49228e3-a4ec-47b8-9ffd-7fc853c4639e\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld" Sep 08 04:08:12.618449 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.618389 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22674c94-7b80-4ba9-8f07-5a890003c00a-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v\" (UID: \"22674c94-7b80-4ba9-8f07-5a890003c00a\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v" Sep 08 04:08:12.618449 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.618420 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldxb8\" (UniqueName: \"kubernetes.io/projected/22674c94-7b80-4ba9-8f07-5a890003c00a-kube-api-access-ldxb8\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v\" (UID: \"22674c94-7b80-4ba9-8f07-5a890003c00a\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v" Sep 08 04:08:12.618570 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.618450 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e49228e3-a4ec-47b8-9ffd-7fc853c4639e-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld\" (UID: \"e49228e3-a4ec-47b8-9ffd-7fc853c4639e\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld" Sep 08 04:08:12.618741 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.618726 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e49228e3-a4ec-47b8-9ffd-7fc853c4639e-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld\" (UID: \"e49228e3-a4ec-47b8-9ffd-7fc853c4639e\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld" Sep 08 04:08:12.618792 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.618767 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e49228e3-a4ec-47b8-9ffd-7fc853c4639e-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld\" (UID: \"e49228e3-a4ec-47b8-9ffd-7fc853c4639e\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld" Sep 08 04:08:12.626552 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.626527 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h78wf\" (UniqueName: \"kubernetes.io/projected/e49228e3-a4ec-47b8-9ffd-7fc853c4639e-kube-api-access-h78wf\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld\" (UID: \"e49228e3-a4ec-47b8-9ffd-7fc853c4639e\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld" Sep 08 04:08:12.678824 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.678797 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld" Sep 08 04:08:12.720051 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.719957 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkd9q\" (UniqueName: \"kubernetes.io/projected/65a09edd-6bcb-480d-a0aa-2eb8c5444f5e-kube-api-access-zkd9q\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr\" (UID: \"65a09edd-6bcb-480d-a0aa-2eb8c5444f5e\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr" Sep 08 04:08:12.720051 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.720002 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65a09edd-6bcb-480d-a0aa-2eb8c5444f5e-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr\" (UID: \"65a09edd-6bcb-480d-a0aa-2eb8c5444f5e\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr" Sep 08 04:08:12.720247 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.720124 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22674c94-7b80-4ba9-8f07-5a890003c00a-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v\" (UID: \"22674c94-7b80-4ba9-8f07-5a890003c00a\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v" Sep 08 04:08:12.720247 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.720175 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65a09edd-6bcb-480d-a0aa-2eb8c5444f5e-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr\" (UID: \"65a09edd-6bcb-480d-a0aa-2eb8c5444f5e\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr" Sep 08 04:08:12.720247 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.720230 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22674c94-7b80-4ba9-8f07-5a890003c00a-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v\" (UID: \"22674c94-7b80-4ba9-8f07-5a890003c00a\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v" Sep 08 04:08:12.720398 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.720255 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldxb8\" (UniqueName: \"kubernetes.io/projected/22674c94-7b80-4ba9-8f07-5a890003c00a-kube-api-access-ldxb8\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v\" (UID: \"22674c94-7b80-4ba9-8f07-5a890003c00a\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v" Sep 08 04:08:12.720605 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.720579 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22674c94-7b80-4ba9-8f07-5a890003c00a-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v\" (UID: \"22674c94-7b80-4ba9-8f07-5a890003c00a\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v" Sep 08 04:08:12.720605 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.720595 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22674c94-7b80-4ba9-8f07-5a890003c00a-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v\" (UID: \"22674c94-7b80-4ba9-8f07-5a890003c00a\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v" Sep 08 04:08:12.729260 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.729203 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldxb8\" (UniqueName: \"kubernetes.io/projected/22674c94-7b80-4ba9-8f07-5a890003c00a-kube-api-access-ldxb8\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v\" (UID: \"22674c94-7b80-4ba9-8f07-5a890003c00a\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v" Sep 08 04:08:12.782663 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.782610 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v" Sep 08 04:08:12.820940 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.820907 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkd9q\" (UniqueName: \"kubernetes.io/projected/65a09edd-6bcb-480d-a0aa-2eb8c5444f5e-kube-api-access-zkd9q\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr\" (UID: \"65a09edd-6bcb-480d-a0aa-2eb8c5444f5e\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr" Sep 08 04:08:12.821082 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.820966 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65a09edd-6bcb-480d-a0aa-2eb8c5444f5e-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr\" (UID: \"65a09edd-6bcb-480d-a0aa-2eb8c5444f5e\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr" Sep 08 04:08:12.821082 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.821070 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65a09edd-6bcb-480d-a0aa-2eb8c5444f5e-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr\" (UID: \"65a09edd-6bcb-480d-a0aa-2eb8c5444f5e\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr" Sep 08 04:08:12.821401 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.821377 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65a09edd-6bcb-480d-a0aa-2eb8c5444f5e-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr\" (UID: \"65a09edd-6bcb-480d-a0aa-2eb8c5444f5e\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr" Sep 08 04:08:12.821526 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.821460 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65a09edd-6bcb-480d-a0aa-2eb8c5444f5e-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr\" (UID: \"65a09edd-6bcb-480d-a0aa-2eb8c5444f5e\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr" Sep 08 04:08:12.829280 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.829257 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkd9q\" (UniqueName: \"kubernetes.io/projected/65a09edd-6bcb-480d-a0aa-2eb8c5444f5e-kube-api-access-zkd9q\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr\" (UID: \"65a09edd-6bcb-480d-a0aa-2eb8c5444f5e\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr" Sep 08 04:08:12.867254 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.867227 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g"] Sep 08 04:08:12.868612 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:08:12.868583 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07fe3c21_bee0_41a0_b6fb_31eaa110a2c1.slice/crio-1d26efa77e816d00eaa0cc8e1d8bb6c208cbc2440306e9ee3fb9b91a395a4d28 WatchSource:0}: Error finding container 1d26efa77e816d00eaa0cc8e1d8bb6c208cbc2440306e9ee3fb9b91a395a4d28: Status 404 returned error can't find the container with id 1d26efa77e816d00eaa0cc8e1d8bb6c208cbc2440306e9ee3fb9b91a395a4d28 Sep 08 04:08:12.885279 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.885255 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr" Sep 08 04:08:12.962289 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:12.962255 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld"] Sep 08 04:08:13.059720 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:13.059694 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v"] Sep 08 04:08:13.060454 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:08:13.060430 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22674c94_7b80_4ba9_8f07_5a890003c00a.slice/crio-e27d333342ef8acb94a2fbbbb92dcb4e8702d55d80e0f4534e162484b3cfd108 WatchSource:0}: Error finding container e27d333342ef8acb94a2fbbbb92dcb4e8702d55d80e0f4534e162484b3cfd108: Status 404 returned error can't find the container with id e27d333342ef8acb94a2fbbbb92dcb4e8702d55d80e0f4534e162484b3cfd108 Sep 08 04:08:13.162717 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:13.162695 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr"] Sep 08 04:08:13.163863 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:08:13.163838 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a09edd_6bcb_480d_a0aa_2eb8c5444f5e.slice/crio-b20a130761568d8898e73e7fd136ff7c656a4cb55a8f60859d6312ec23d4257f WatchSource:0}: Error finding container b20a130761568d8898e73e7fd136ff7c656a4cb55a8f60859d6312ec23d4257f: Status 404 returned error can't find the container with id b20a130761568d8898e73e7fd136ff7c656a4cb55a8f60859d6312ec23d4257f Sep 08 04:08:13.457125 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:13.457069 2574 generic.go:356] "Generic (PLEG): container finished" podID="e49228e3-a4ec-47b8-9ffd-7fc853c4639e" containerID="8cf86a4de3b27cb1171405b5d3fa2d834ce31ee23ee200ba8706009e7a0ab4fd" exitCode=0 Sep 08 04:08:13.457478 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:13.457161 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld" event={"ID":"e49228e3-a4ec-47b8-9ffd-7fc853c4639e","Type":"ContainerDied","Data":"8cf86a4de3b27cb1171405b5d3fa2d834ce31ee23ee200ba8706009e7a0ab4fd"} Sep 08 04:08:13.457478 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:13.457213 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld" event={"ID":"e49228e3-a4ec-47b8-9ffd-7fc853c4639e","Type":"ContainerStarted","Data":"8089e8b1fd540d2636b681ea140f5a61cb007fc4a4e28f078efc6538597ee71c"} Sep 08 04:08:13.458808 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:13.458787 2574 generic.go:356] "Generic (PLEG): container finished" podID="65a09edd-6bcb-480d-a0aa-2eb8c5444f5e" containerID="5ff250e0bffe22a16cefbb399cd8937105daf3a009ab848a1eb8eecf29d420c3" exitCode=0 Sep 08 04:08:13.458905 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:13.458865 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr" event={"ID":"65a09edd-6bcb-480d-a0aa-2eb8c5444f5e","Type":"ContainerDied","Data":"5ff250e0bffe22a16cefbb399cd8937105daf3a009ab848a1eb8eecf29d420c3"} Sep 08 04:08:13.458970 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:13.458914 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr" event={"ID":"65a09edd-6bcb-480d-a0aa-2eb8c5444f5e","Type":"ContainerStarted","Data":"b20a130761568d8898e73e7fd136ff7c656a4cb55a8f60859d6312ec23d4257f"} Sep 08 04:08:13.460482 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:13.460465 2574 generic.go:356] "Generic (PLEG): container finished" podID="07fe3c21-bee0-41a0-b6fb-31eaa110a2c1" containerID="7dd98beffc9efd76f07b89c7099fdc7bd51678118a0c7c3ba6cc03f815aff64e" exitCode=0 Sep 08 04:08:13.460570 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:13.460550 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g" event={"ID":"07fe3c21-bee0-41a0-b6fb-31eaa110a2c1","Type":"ContainerDied","Data":"7dd98beffc9efd76f07b89c7099fdc7bd51678118a0c7c3ba6cc03f815aff64e"} Sep 08 04:08:13.460624 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:13.460608 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g" event={"ID":"07fe3c21-bee0-41a0-b6fb-31eaa110a2c1","Type":"ContainerStarted","Data":"1d26efa77e816d00eaa0cc8e1d8bb6c208cbc2440306e9ee3fb9b91a395a4d28"} Sep 08 04:08:13.462330 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:13.462309 2574 generic.go:356] "Generic (PLEG): container finished" podID="22674c94-7b80-4ba9-8f07-5a890003c00a" containerID="dacf7d7cd6aa5b1b42dce813cea8fb821d85d33e9fd4e1c38a8fd5abdc5f64a5" exitCode=0 Sep 08 04:08:13.462621 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:13.462357 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v" event={"ID":"22674c94-7b80-4ba9-8f07-5a890003c00a","Type":"ContainerDied","Data":"dacf7d7cd6aa5b1b42dce813cea8fb821d85d33e9fd4e1c38a8fd5abdc5f64a5"} Sep 08 04:08:13.462621 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:13.462385 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v" event={"ID":"22674c94-7b80-4ba9-8f07-5a890003c00a","Type":"ContainerStarted","Data":"e27d333342ef8acb94a2fbbbb92dcb4e8702d55d80e0f4534e162484b3cfd108"} Sep 08 04:08:15.474555 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:15.474521 2574 generic.go:356] "Generic (PLEG): container finished" podID="e49228e3-a4ec-47b8-9ffd-7fc853c4639e" containerID="fedf3a026eb865a55436b8f6511e1e834a63bd6e4c67cfb587afc2c6e43bacd8" exitCode=0 Sep 08 04:08:15.474878 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:15.474595 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld" event={"ID":"e49228e3-a4ec-47b8-9ffd-7fc853c4639e","Type":"ContainerDied","Data":"fedf3a026eb865a55436b8f6511e1e834a63bd6e4c67cfb587afc2c6e43bacd8"} Sep 08 04:08:15.476652 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:15.476627 2574 generic.go:356] "Generic (PLEG): container finished" podID="65a09edd-6bcb-480d-a0aa-2eb8c5444f5e" containerID="be322bb79ca6f4a635e9a8d6293de66af09fd4ff2bb433ae8bc2f3dfe8a0db8e" exitCode=0 Sep 08 04:08:15.476746 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:15.476656 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr" event={"ID":"65a09edd-6bcb-480d-a0aa-2eb8c5444f5e","Type":"ContainerDied","Data":"be322bb79ca6f4a635e9a8d6293de66af09fd4ff2bb433ae8bc2f3dfe8a0db8e"} Sep 08 04:08:15.478679 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:15.478661 2574 generic.go:356] "Generic (PLEG): container finished" podID="07fe3c21-bee0-41a0-b6fb-31eaa110a2c1" containerID="6987e02e9c72d0658bb4788c4a74c78ec79dba2c0c168947a0b95497bc6960b2" exitCode=0 Sep 08 04:08:15.478771 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:15.478736 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g" event={"ID":"07fe3c21-bee0-41a0-b6fb-31eaa110a2c1","Type":"ContainerDied","Data":"6987e02e9c72d0658bb4788c4a74c78ec79dba2c0c168947a0b95497bc6960b2"} Sep 08 04:08:15.480766 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:15.480746 2574 generic.go:356] "Generic (PLEG): container finished" podID="22674c94-7b80-4ba9-8f07-5a890003c00a" containerID="a4d448a2971cfa874c19908eeeb93d2ff6bee960206284da4395a78c8b6b262a" exitCode=0 Sep 08 04:08:15.480847 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:15.480827 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v" event={"ID":"22674c94-7b80-4ba9-8f07-5a890003c00a","Type":"ContainerDied","Data":"a4d448a2971cfa874c19908eeeb93d2ff6bee960206284da4395a78c8b6b262a"} Sep 08 04:08:16.488126 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:16.488098 2574 generic.go:356] "Generic (PLEG): container finished" podID="e49228e3-a4ec-47b8-9ffd-7fc853c4639e" containerID="240ebd1b7f3b38acaaf6cbdb2e29a6aced3d17c2fb8c140a23616cc56519395d" exitCode=0 Sep 08 04:08:16.488489 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:16.488182 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld" event={"ID":"e49228e3-a4ec-47b8-9ffd-7fc853c4639e","Type":"ContainerDied","Data":"240ebd1b7f3b38acaaf6cbdb2e29a6aced3d17c2fb8c140a23616cc56519395d"} Sep 08 04:08:16.490390 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:16.490368 2574 generic.go:356] "Generic (PLEG): container finished" podID="65a09edd-6bcb-480d-a0aa-2eb8c5444f5e" containerID="e5ba765336b278647d203a02dd147ab8ffa7e06d410f0cc724a27d6e3a261e68" exitCode=0 Sep 08 04:08:16.490482 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:16.490415 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr" event={"ID":"65a09edd-6bcb-480d-a0aa-2eb8c5444f5e","Type":"ContainerDied","Data":"e5ba765336b278647d203a02dd147ab8ffa7e06d410f0cc724a27d6e3a261e68"} Sep 08 04:08:16.492753 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:16.492732 2574 generic.go:356] "Generic (PLEG): container finished" podID="07fe3c21-bee0-41a0-b6fb-31eaa110a2c1" containerID="002e5347a6ba70f945e852d8e4a9f218496e0719a15fdcb7044c74dfbf7e4c6c" exitCode=0 Sep 08 04:08:16.492830 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:16.492813 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g" event={"ID":"07fe3c21-bee0-41a0-b6fb-31eaa110a2c1","Type":"ContainerDied","Data":"002e5347a6ba70f945e852d8e4a9f218496e0719a15fdcb7044c74dfbf7e4c6c"} Sep 08 04:08:16.494889 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:16.494869 2574 generic.go:356] "Generic (PLEG): container finished" podID="22674c94-7b80-4ba9-8f07-5a890003c00a" containerID="f202af220ab54727d7a26869ce169b4d40116e10d56a1bf3e3dbf6566020fefc" exitCode=0 Sep 08 04:08:16.494969 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:16.494921 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v" event={"ID":"22674c94-7b80-4ba9-8f07-5a890003c00a","Type":"ContainerDied","Data":"f202af220ab54727d7a26869ce169b4d40116e10d56a1bf3e3dbf6566020fefc"} Sep 08 04:08:17.670280 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.670254 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v" Sep 08 04:08:17.673178 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.673157 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g" Sep 08 04:08:17.719554 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.719532 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld" Sep 08 04:08:17.722457 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.722433 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr" Sep 08 04:08:17.760982 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.760913 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h78wf\" (UniqueName: \"kubernetes.io/projected/e49228e3-a4ec-47b8-9ffd-7fc853c4639e-kube-api-access-h78wf\") pod \"e49228e3-a4ec-47b8-9ffd-7fc853c4639e\" (UID: \"e49228e3-a4ec-47b8-9ffd-7fc853c4639e\") " Sep 08 04:08:17.760982 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.760958 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkd9q\" (UniqueName: \"kubernetes.io/projected/65a09edd-6bcb-480d-a0aa-2eb8c5444f5e-kube-api-access-zkd9q\") pod \"65a09edd-6bcb-480d-a0aa-2eb8c5444f5e\" (UID: \"65a09edd-6bcb-480d-a0aa-2eb8c5444f5e\") " Sep 08 04:08:17.761154 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.761003 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldxb8\" (UniqueName: \"kubernetes.io/projected/22674c94-7b80-4ba9-8f07-5a890003c00a-kube-api-access-ldxb8\") pod \"22674c94-7b80-4ba9-8f07-5a890003c00a\" (UID: \"22674c94-7b80-4ba9-8f07-5a890003c00a\") " Sep 08 04:08:17.761154 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.761033 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07fe3c21-bee0-41a0-b6fb-31eaa110a2c1-bundle\") pod \"07fe3c21-bee0-41a0-b6fb-31eaa110a2c1\" (UID: \"07fe3c21-bee0-41a0-b6fb-31eaa110a2c1\") " Sep 08 04:08:17.761154 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.761057 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65a09edd-6bcb-480d-a0aa-2eb8c5444f5e-bundle\") pod \"65a09edd-6bcb-480d-a0aa-2eb8c5444f5e\" (UID: \"65a09edd-6bcb-480d-a0aa-2eb8c5444f5e\") " Sep 08 04:08:17.761154 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.761082 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e49228e3-a4ec-47b8-9ffd-7fc853c4639e-util\") pod \"e49228e3-a4ec-47b8-9ffd-7fc853c4639e\" (UID: \"e49228e3-a4ec-47b8-9ffd-7fc853c4639e\") " Sep 08 04:08:17.761154 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.761126 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65a09edd-6bcb-480d-a0aa-2eb8c5444f5e-util\") pod \"65a09edd-6bcb-480d-a0aa-2eb8c5444f5e\" (UID: \"65a09edd-6bcb-480d-a0aa-2eb8c5444f5e\") " Sep 08 04:08:17.761412 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.761176 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e49228e3-a4ec-47b8-9ffd-7fc853c4639e-bundle\") pod \"e49228e3-a4ec-47b8-9ffd-7fc853c4639e\" (UID: \"e49228e3-a4ec-47b8-9ffd-7fc853c4639e\") " Sep 08 04:08:17.761412 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.761199 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22674c94-7b80-4ba9-8f07-5a890003c00a-bundle\") pod \"22674c94-7b80-4ba9-8f07-5a890003c00a\" (UID: \"22674c94-7b80-4ba9-8f07-5a890003c00a\") " Sep 08 04:08:17.761412 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.761221 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22674c94-7b80-4ba9-8f07-5a890003c00a-util\") pod \"22674c94-7b80-4ba9-8f07-5a890003c00a\" (UID: \"22674c94-7b80-4ba9-8f07-5a890003c00a\") " Sep 08 04:08:17.761412 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.761250 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvmcq\" (UniqueName: \"kubernetes.io/projected/07fe3c21-bee0-41a0-b6fb-31eaa110a2c1-kube-api-access-lvmcq\") pod \"07fe3c21-bee0-41a0-b6fb-31eaa110a2c1\" (UID: \"07fe3c21-bee0-41a0-b6fb-31eaa110a2c1\") " Sep 08 04:08:17.761412 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.761308 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07fe3c21-bee0-41a0-b6fb-31eaa110a2c1-util\") pod \"07fe3c21-bee0-41a0-b6fb-31eaa110a2c1\" (UID: \"07fe3c21-bee0-41a0-b6fb-31eaa110a2c1\") " Sep 08 04:08:17.766388 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.761975 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a09edd-6bcb-480d-a0aa-2eb8c5444f5e-bundle" (OuterVolumeSpecName: "bundle") pod "65a09edd-6bcb-480d-a0aa-2eb8c5444f5e" (UID: "65a09edd-6bcb-480d-a0aa-2eb8c5444f5e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:08:17.766388 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.762047 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fe3c21-bee0-41a0-b6fb-31eaa110a2c1-bundle" (OuterVolumeSpecName: "bundle") pod "07fe3c21-bee0-41a0-b6fb-31eaa110a2c1" (UID: "07fe3c21-bee0-41a0-b6fb-31eaa110a2c1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:08:17.766388 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.762057 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49228e3-a4ec-47b8-9ffd-7fc853c4639e-bundle" (OuterVolumeSpecName: "bundle") pod "e49228e3-a4ec-47b8-9ffd-7fc853c4639e" (UID: "e49228e3-a4ec-47b8-9ffd-7fc853c4639e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:08:17.766388 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.762546 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22674c94-7b80-4ba9-8f07-5a890003c00a-bundle" (OuterVolumeSpecName: "bundle") pod "22674c94-7b80-4ba9-8f07-5a890003c00a" (UID: "22674c94-7b80-4ba9-8f07-5a890003c00a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:08:17.766388 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.763720 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a09edd-6bcb-480d-a0aa-2eb8c5444f5e-kube-api-access-zkd9q" (OuterVolumeSpecName: "kube-api-access-zkd9q") pod "65a09edd-6bcb-480d-a0aa-2eb8c5444f5e" (UID: "65a09edd-6bcb-480d-a0aa-2eb8c5444f5e"). InnerVolumeSpecName "kube-api-access-zkd9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:08:17.766875 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.766840 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49228e3-a4ec-47b8-9ffd-7fc853c4639e-kube-api-access-h78wf" (OuterVolumeSpecName: "kube-api-access-h78wf") pod "e49228e3-a4ec-47b8-9ffd-7fc853c4639e" (UID: "e49228e3-a4ec-47b8-9ffd-7fc853c4639e"). InnerVolumeSpecName "kube-api-access-h78wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:08:17.767952 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.767924 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22674c94-7b80-4ba9-8f07-5a890003c00a-kube-api-access-ldxb8" (OuterVolumeSpecName: "kube-api-access-ldxb8") pod "22674c94-7b80-4ba9-8f07-5a890003c00a" (UID: "22674c94-7b80-4ba9-8f07-5a890003c00a"). InnerVolumeSpecName "kube-api-access-ldxb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:08:17.768350 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.768316 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07fe3c21-bee0-41a0-b6fb-31eaa110a2c1-kube-api-access-lvmcq" (OuterVolumeSpecName: "kube-api-access-lvmcq") pod "07fe3c21-bee0-41a0-b6fb-31eaa110a2c1" (UID: "07fe3c21-bee0-41a0-b6fb-31eaa110a2c1"). InnerVolumeSpecName "kube-api-access-lvmcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:08:17.769804 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.769776 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49228e3-a4ec-47b8-9ffd-7fc853c4639e-util" (OuterVolumeSpecName: "util") pod "e49228e3-a4ec-47b8-9ffd-7fc853c4639e" (UID: "e49228e3-a4ec-47b8-9ffd-7fc853c4639e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:08:17.770466 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.770444 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fe3c21-bee0-41a0-b6fb-31eaa110a2c1-util" (OuterVolumeSpecName: "util") pod "07fe3c21-bee0-41a0-b6fb-31eaa110a2c1" (UID: "07fe3c21-bee0-41a0-b6fb-31eaa110a2c1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:08:17.771835 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.771812 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22674c94-7b80-4ba9-8f07-5a890003c00a-util" (OuterVolumeSpecName: "util") pod "22674c94-7b80-4ba9-8f07-5a890003c00a" (UID: "22674c94-7b80-4ba9-8f07-5a890003c00a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:08:17.772329 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.772306 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a09edd-6bcb-480d-a0aa-2eb8c5444f5e-util" (OuterVolumeSpecName: "util") pod "65a09edd-6bcb-480d-a0aa-2eb8c5444f5e" (UID: "65a09edd-6bcb-480d-a0aa-2eb8c5444f5e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:08:17.862468 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.862446 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65a09edd-6bcb-480d-a0aa-2eb8c5444f5e-util\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:08:17.862468 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.862468 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e49228e3-a4ec-47b8-9ffd-7fc853c4639e-bundle\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:08:17.862591 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.862476 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22674c94-7b80-4ba9-8f07-5a890003c00a-bundle\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:08:17.862591 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.862483 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22674c94-7b80-4ba9-8f07-5a890003c00a-util\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:08:17.862591 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.862492 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvmcq\" (UniqueName: \"kubernetes.io/projected/07fe3c21-bee0-41a0-b6fb-31eaa110a2c1-kube-api-access-lvmcq\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:08:17.862591 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.862519 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07fe3c21-bee0-41a0-b6fb-31eaa110a2c1-util\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:08:17.862591 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.862531 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h78wf\" (UniqueName: \"kubernetes.io/projected/e49228e3-a4ec-47b8-9ffd-7fc853c4639e-kube-api-access-h78wf\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:08:17.862591 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.862542 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkd9q\" (UniqueName: \"kubernetes.io/projected/65a09edd-6bcb-480d-a0aa-2eb8c5444f5e-kube-api-access-zkd9q\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:08:17.862591 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.862552 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldxb8\" (UniqueName: \"kubernetes.io/projected/22674c94-7b80-4ba9-8f07-5a890003c00a-kube-api-access-ldxb8\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:08:17.862591 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.862560 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07fe3c21-bee0-41a0-b6fb-31eaa110a2c1-bundle\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:08:17.862591 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.862568 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65a09edd-6bcb-480d-a0aa-2eb8c5444f5e-bundle\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:08:17.862591 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:17.862575 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e49228e3-a4ec-47b8-9ffd-7fc853c4639e-util\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:08:18.507920 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:18.507895 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr" Sep 08 04:08:18.508043 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:18.507898 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bfdrr" event={"ID":"65a09edd-6bcb-480d-a0aa-2eb8c5444f5e","Type":"ContainerDied","Data":"b20a130761568d8898e73e7fd136ff7c656a4cb55a8f60859d6312ec23d4257f"} Sep 08 04:08:18.508082 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:18.508069 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b20a130761568d8898e73e7fd136ff7c656a4cb55a8f60859d6312ec23d4257f" Sep 08 04:08:18.509957 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:18.509932 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g" Sep 08 04:08:18.510088 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:18.509930 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4775g" event={"ID":"07fe3c21-bee0-41a0-b6fb-31eaa110a2c1","Type":"ContainerDied","Data":"1d26efa77e816d00eaa0cc8e1d8bb6c208cbc2440306e9ee3fb9b91a395a4d28"} Sep 08 04:08:18.510088 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:18.510053 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d26efa77e816d00eaa0cc8e1d8bb6c208cbc2440306e9ee3fb9b91a395a4d28" Sep 08 04:08:18.511903 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:18.511887 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v" Sep 08 04:08:18.512012 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:18.511927 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c2bn5v" event={"ID":"22674c94-7b80-4ba9-8f07-5a890003c00a","Type":"ContainerDied","Data":"e27d333342ef8acb94a2fbbbb92dcb4e8702d55d80e0f4534e162484b3cfd108"} Sep 08 04:08:18.512012 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:18.511963 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e27d333342ef8acb94a2fbbbb92dcb4e8702d55d80e0f4534e162484b3cfd108" Sep 08 04:08:18.514193 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:18.514172 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld" event={"ID":"e49228e3-a4ec-47b8-9ffd-7fc853c4639e","Type":"ContainerDied","Data":"8089e8b1fd540d2636b681ea140f5a61cb007fc4a4e28f078efc6538597ee71c"} Sep 08 04:08:18.514356 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:18.514205 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8089e8b1fd540d2636b681ea140f5a61cb007fc4a4e28f078efc6538597ee71c" Sep 08 04:08:18.514356 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:18.514231 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539wfld" Sep 08 04:08:22.686727 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:22.686695 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:25.411251 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:25.411224 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:27.196722 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:27.196683 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:27.458132 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:27.458048 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:27.625905 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:27.625853 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:28.195802 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:28.195754 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:28.735277 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:28.735251 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:29.681643 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:29.681606 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:31.396725 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:31.396696 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:31.820115 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:31.820081 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:33.657520 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:33.657461 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:33.754827 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:33.754800 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:35.215304 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:35.215275 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:35.316146 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:35.316113 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:38.654604 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:38.654574 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:38.912415 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:38.912315 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:39.013405 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:39.013354 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:39.118340 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:39.118290 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:42.829972 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:42.829933 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:42.927354 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:42.927323 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:43.027215 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:43.027190 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:43.089842 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:43.089768 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:48.194975 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:48.194936 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:51.669850 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:51.669789 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:51.778413 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:51.778386 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:51.882427 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:51.882400 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:54.417782 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:54.417737 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:54.569596 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:54.569566 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:57.480083 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:57.480051 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:08:57.578284 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:08:57.578253 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:09:02.056596 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:09:02.056562 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:09:02.147518 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:09:02.147460 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:09:02.243293 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:09:02.243269 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:09:03.091572 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:09:03.091539 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:09:03.191947 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:09:03.191919 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:09:10.815553 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:09:10.815514 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:09:10.917957 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:09:10.917904 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:09:12.312043 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:09:12.312011 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:09:12.365263 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:09:12.365234 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:09:18.507219 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:09:18.507175 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:09:18.953643 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:09:18.953485 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:09:32.053211 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:09:32.053156 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:09:32.249173 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:09:32.249126 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:09:32.349801 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:09:32.349716 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:09:32.897972 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:09:32.897942 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:09:33.007886 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:09:33.007860 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:09:40.787421 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:09:40.787374 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:09:49.536014 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:09:49.535934 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:09:49.640377 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:09:49.640335 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:09:50.042266 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:09:50.042219 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:10:02.721454 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:10:02.721396 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:10:03.166725 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:10:03.166690 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:10:10.161822 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:10:10.161793 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:10:10.262628 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:10:10.262595 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:10:10.361388 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:10:10.361360 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:10:10.462715 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:10:10.462652 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:10:10.630035 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:10:10.630003 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:10:14.403173 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:10:14.403136 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:10:14.499902 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:10:14.499869 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:10:45.693589 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:10:45.693550 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:10:47.369960 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:10:47.369928 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:10:47.654781 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:10:47.654702 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:10:47.751391 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:10:47.751359 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:10:47.829407 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:10:47.829376 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:10:57.760538 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:10:57.760422 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:10:57.857909 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:10:57.857882 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:11:11.570293 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570262 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l"] Sep 08 04:11:11.570735 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570577 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22674c94-7b80-4ba9-8f07-5a890003c00a" containerName="util" Sep 08 04:11:11.570735 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570590 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="22674c94-7b80-4ba9-8f07-5a890003c00a" containerName="util" Sep 08 04:11:11.570735 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570600 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07fe3c21-bee0-41a0-b6fb-31eaa110a2c1" containerName="extract" Sep 08 04:11:11.570735 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570606 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="07fe3c21-bee0-41a0-b6fb-31eaa110a2c1" containerName="extract" Sep 08 04:11:11.570735 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570612 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65a09edd-6bcb-480d-a0aa-2eb8c5444f5e" containerName="util" Sep 08 04:11:11.570735 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570620 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="65a09edd-6bcb-480d-a0aa-2eb8c5444f5e" containerName="util" Sep 08 04:11:11.570735 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570630 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22674c94-7b80-4ba9-8f07-5a890003c00a" containerName="pull" Sep 08 04:11:11.570735 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570635 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="22674c94-7b80-4ba9-8f07-5a890003c00a" containerName="pull" Sep 08 04:11:11.570735 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570644 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65a09edd-6bcb-480d-a0aa-2eb8c5444f5e" containerName="pull" Sep 08 04:11:11.570735 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570649 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="65a09edd-6bcb-480d-a0aa-2eb8c5444f5e" containerName="pull" Sep 08 04:11:11.570735 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570658 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07fe3c21-bee0-41a0-b6fb-31eaa110a2c1" containerName="util" Sep 08 04:11:11.570735 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570663 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="07fe3c21-bee0-41a0-b6fb-31eaa110a2c1" containerName="util" Sep 08 04:11:11.570735 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570674 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e49228e3-a4ec-47b8-9ffd-7fc853c4639e" containerName="pull" Sep 08 04:11:11.570735 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570679 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e49228e3-a4ec-47b8-9ffd-7fc853c4639e" containerName="pull" Sep 08 04:11:11.570735 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570687 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e49228e3-a4ec-47b8-9ffd-7fc853c4639e" containerName="extract" Sep 08 04:11:11.570735 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570692 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e49228e3-a4ec-47b8-9ffd-7fc853c4639e" containerName="extract" Sep 08 04:11:11.570735 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570698 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22674c94-7b80-4ba9-8f07-5a890003c00a" containerName="extract" Sep 08 04:11:11.570735 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570703 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="22674c94-7b80-4ba9-8f07-5a890003c00a" containerName="extract" Sep 08 04:11:11.570735 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570713 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07fe3c21-bee0-41a0-b6fb-31eaa110a2c1" containerName="pull" Sep 08 04:11:11.570735 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570717 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="07fe3c21-bee0-41a0-b6fb-31eaa110a2c1" containerName="pull" Sep 08 04:11:11.570735 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570723 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65a09edd-6bcb-480d-a0aa-2eb8c5444f5e" containerName="extract" Sep 08 04:11:11.570735 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570728 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="65a09edd-6bcb-480d-a0aa-2eb8c5444f5e" containerName="extract" Sep 08 04:11:11.570735 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570735 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e49228e3-a4ec-47b8-9ffd-7fc853c4639e" containerName="util" Sep 08 04:11:11.570735 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570740 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e49228e3-a4ec-47b8-9ffd-7fc853c4639e" containerName="util" Sep 08 04:11:11.571482 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570794 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e49228e3-a4ec-47b8-9ffd-7fc853c4639e" containerName="extract" Sep 08 04:11:11.571482 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570803 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="22674c94-7b80-4ba9-8f07-5a890003c00a" containerName="extract" Sep 08 04:11:11.571482 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570810 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="07fe3c21-bee0-41a0-b6fb-31eaa110a2c1" containerName="extract" Sep 08 04:11:11.571482 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.570815 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="65a09edd-6bcb-480d-a0aa-2eb8c5444f5e" containerName="extract" Sep 08 04:11:11.573757 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.573740 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l" Sep 08 04:11:11.577413 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.577391 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 08 04:11:11.577659 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.577638 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ts7g5\"" Sep 08 04:11:11.577770 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.577665 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 08 04:11:11.577770 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.577641 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 08 04:11:11.584611 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.584591 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l"] Sep 08 04:11:11.592708 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.592683 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:11:11.715643 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.715610 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmrth\" (UniqueName: \"kubernetes.io/projected/fad15175-3c67-4e80-8416-a1087cde7b99-kube-api-access-xmrth\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l\" (UID: \"fad15175-3c67-4e80-8416-a1087cde7b99\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l" Sep 08 04:11:11.715643 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.715647 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fad15175-3c67-4e80-8416-a1087cde7b99-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l\" (UID: \"fad15175-3c67-4e80-8416-a1087cde7b99\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l" Sep 08 04:11:11.715825 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.715664 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fad15175-3c67-4e80-8416-a1087cde7b99-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l\" (UID: \"fad15175-3c67-4e80-8416-a1087cde7b99\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l" Sep 08 04:11:11.715825 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.715715 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fad15175-3c67-4e80-8416-a1087cde7b99-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l\" (UID: \"fad15175-3c67-4e80-8416-a1087cde7b99\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l" Sep 08 04:11:11.715825 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.715752 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fad15175-3c67-4e80-8416-a1087cde7b99-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l\" (UID: \"fad15175-3c67-4e80-8416-a1087cde7b99\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l" Sep 08 04:11:11.715825 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.715767 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fad15175-3c67-4e80-8416-a1087cde7b99-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l\" (UID: \"fad15175-3c67-4e80-8416-a1087cde7b99\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l" Sep 08 04:11:11.817030 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.816999 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmrth\" (UniqueName: \"kubernetes.io/projected/fad15175-3c67-4e80-8416-a1087cde7b99-kube-api-access-xmrth\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l\" (UID: \"fad15175-3c67-4e80-8416-a1087cde7b99\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l" Sep 08 04:11:11.817194 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.817040 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fad15175-3c67-4e80-8416-a1087cde7b99-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l\" (UID: \"fad15175-3c67-4e80-8416-a1087cde7b99\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l" Sep 08 04:11:11.817194 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.817059 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fad15175-3c67-4e80-8416-a1087cde7b99-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l\" (UID: \"fad15175-3c67-4e80-8416-a1087cde7b99\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l" Sep 08 04:11:11.817194 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.817085 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fad15175-3c67-4e80-8416-a1087cde7b99-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l\" (UID: \"fad15175-3c67-4e80-8416-a1087cde7b99\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l" Sep 08 04:11:11.817194 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.817116 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fad15175-3c67-4e80-8416-a1087cde7b99-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l\" (UID: \"fad15175-3c67-4e80-8416-a1087cde7b99\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l" Sep 08 04:11:11.817194 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.817132 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fad15175-3c67-4e80-8416-a1087cde7b99-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l\" (UID: \"fad15175-3c67-4e80-8416-a1087cde7b99\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l" Sep 08 04:11:11.817444 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.817426 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fad15175-3c67-4e80-8416-a1087cde7b99-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l\" (UID: \"fad15175-3c67-4e80-8416-a1087cde7b99\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l" Sep 08 04:11:11.817526 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.817456 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fad15175-3c67-4e80-8416-a1087cde7b99-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l\" (UID: \"fad15175-3c67-4e80-8416-a1087cde7b99\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l" Sep 08 04:11:11.817619 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.817599 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fad15175-3c67-4e80-8416-a1087cde7b99-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l\" (UID: \"fad15175-3c67-4e80-8416-a1087cde7b99\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l" Sep 08 04:11:11.819442 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.819412 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fad15175-3c67-4e80-8416-a1087cde7b99-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l\" (UID: \"fad15175-3c67-4e80-8416-a1087cde7b99\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l" Sep 08 04:11:11.819711 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.819691 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fad15175-3c67-4e80-8416-a1087cde7b99-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l\" (UID: \"fad15175-3c67-4e80-8416-a1087cde7b99\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l" Sep 08 04:11:11.825281 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.825232 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmrth\" (UniqueName: \"kubernetes.io/projected/fad15175-3c67-4e80-8416-a1087cde7b99-kube-api-access-xmrth\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l\" (UID: \"fad15175-3c67-4e80-8416-a1087cde7b99\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l" Sep 08 04:11:11.884799 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:11.884772 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l" Sep 08 04:11:12.172805 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:12.172768 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l"] Sep 08 04:11:12.174352 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:11:12.174328 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad15175_3c67_4e80_8416_a1087cde7b99.slice/crio-05e19b27da048adacc01b4eed9bcf175e4d078558b8301cec5eb464fe7ae4380 WatchSource:0}: Error finding container 05e19b27da048adacc01b4eed9bcf175e4d078558b8301cec5eb464fe7ae4380: Status 404 returned error can't find the container with id 05e19b27da048adacc01b4eed9bcf175e4d078558b8301cec5eb464fe7ae4380 Sep 08 04:11:12.176049 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:12.176031 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 04:11:12.356926 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:12.356889 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l" event={"ID":"fad15175-3c67-4e80-8416-a1087cde7b99","Type":"ContainerStarted","Data":"05e19b27da048adacc01b4eed9bcf175e4d078558b8301cec5eb464fe7ae4380"} Sep 08 04:11:13.363178 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:13.363147 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l" event={"ID":"fad15175-3c67-4e80-8416-a1087cde7b99","Type":"ContainerStarted","Data":"35724b8ddb1cb0669249d091113314c4c952c07ffcc95b0eef3cfae42a2da9f8"} Sep 08 04:11:13.381772 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:13.381703 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l" podStartSLOduration=1.271005055 podStartE2EDuration="2.38168676s" totalImagesPullingTime="1.110681705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:11:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:11:13.380346523 +0000 UTC m=+647.302807282" watchObservedRunningTime="2026-09-08 04:11:13.38168676 +0000 UTC m=+647.304147499" Sep 08 04:11:21.885463 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:21.885427 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l" Sep 08 04:11:21.885878 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:21.885497 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l" Sep 08 04:11:21.898114 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:21.898088 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l" Sep 08 04:11:22.418883 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:22.418852 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l" Sep 08 04:11:30.750021 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:30.749972 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx"] Sep 08 04:11:30.753331 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:30.753298 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx" Sep 08 04:11:30.755806 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:30.755782 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 08 04:11:30.761683 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:30.761658 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx"] Sep 08 04:11:30.779456 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:30.779433 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:11:30.871859 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:30.871828 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klsp4\" (UniqueName: \"kubernetes.io/projected/a342471e-e7cc-4a0f-9f71-be694b5c88b5-kube-api-access-klsp4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fg2fx\" (UID: \"a342471e-e7cc-4a0f-9f71-be694b5c88b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx" Sep 08 04:11:30.871996 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:30.871864 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a342471e-e7cc-4a0f-9f71-be694b5c88b5-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fg2fx\" (UID: \"a342471e-e7cc-4a0f-9f71-be694b5c88b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx" Sep 08 04:11:30.871996 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:30.871939 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a342471e-e7cc-4a0f-9f71-be694b5c88b5-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fg2fx\" (UID: \"a342471e-e7cc-4a0f-9f71-be694b5c88b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx" Sep 08 04:11:30.872114 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:30.872012 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a342471e-e7cc-4a0f-9f71-be694b5c88b5-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fg2fx\" (UID: \"a342471e-e7cc-4a0f-9f71-be694b5c88b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx" Sep 08 04:11:30.872114 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:30.872055 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a342471e-e7cc-4a0f-9f71-be694b5c88b5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fg2fx\" (UID: \"a342471e-e7cc-4a0f-9f71-be694b5c88b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx" Sep 08 04:11:30.872114 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:30.872079 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a342471e-e7cc-4a0f-9f71-be694b5c88b5-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fg2fx\" (UID: \"a342471e-e7cc-4a0f-9f71-be694b5c88b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx" Sep 08 04:11:30.973089 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:30.973054 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a342471e-e7cc-4a0f-9f71-be694b5c88b5-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fg2fx\" (UID: \"a342471e-e7cc-4a0f-9f71-be694b5c88b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx" Sep 08 04:11:30.973223 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:30.973105 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a342471e-e7cc-4a0f-9f71-be694b5c88b5-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fg2fx\" (UID: \"a342471e-e7cc-4a0f-9f71-be694b5c88b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx" Sep 08 04:11:30.973223 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:30.973133 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a342471e-e7cc-4a0f-9f71-be694b5c88b5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fg2fx\" (UID: \"a342471e-e7cc-4a0f-9f71-be694b5c88b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx" Sep 08 04:11:30.973223 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:30.973149 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a342471e-e7cc-4a0f-9f71-be694b5c88b5-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fg2fx\" (UID: \"a342471e-e7cc-4a0f-9f71-be694b5c88b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx" Sep 08 04:11:30.973223 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:30.973178 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klsp4\" (UniqueName: \"kubernetes.io/projected/a342471e-e7cc-4a0f-9f71-be694b5c88b5-kube-api-access-klsp4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fg2fx\" (UID: \"a342471e-e7cc-4a0f-9f71-be694b5c88b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx" Sep 08 04:11:30.973223 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:30.973193 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a342471e-e7cc-4a0f-9f71-be694b5c88b5-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fg2fx\" (UID: \"a342471e-e7cc-4a0f-9f71-be694b5c88b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx" Sep 08 04:11:30.973564 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:30.973528 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a342471e-e7cc-4a0f-9f71-be694b5c88b5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fg2fx\" (UID: \"a342471e-e7cc-4a0f-9f71-be694b5c88b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx" Sep 08 04:11:30.973564 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:30.973559 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a342471e-e7cc-4a0f-9f71-be694b5c88b5-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fg2fx\" (UID: \"a342471e-e7cc-4a0f-9f71-be694b5c88b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx" Sep 08 04:11:30.973706 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:30.973618 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a342471e-e7cc-4a0f-9f71-be694b5c88b5-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fg2fx\" (UID: \"a342471e-e7cc-4a0f-9f71-be694b5c88b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx" Sep 08 04:11:30.975423 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:30.975401 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a342471e-e7cc-4a0f-9f71-be694b5c88b5-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fg2fx\" (UID: \"a342471e-e7cc-4a0f-9f71-be694b5c88b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx" Sep 08 04:11:30.975569 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:30.975551 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a342471e-e7cc-4a0f-9f71-be694b5c88b5-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fg2fx\" (UID: \"a342471e-e7cc-4a0f-9f71-be694b5c88b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx" Sep 08 04:11:30.981341 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:30.981319 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klsp4\" (UniqueName: \"kubernetes.io/projected/a342471e-e7cc-4a0f-9f71-be694b5c88b5-kube-api-access-klsp4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fg2fx\" (UID: \"a342471e-e7cc-4a0f-9f71-be694b5c88b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx" Sep 08 04:11:31.064434 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:31.064369 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx" Sep 08 04:11:31.334010 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:31.333809 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx"] Sep 08 04:11:31.336560 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:11:31.336532 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda342471e_e7cc_4a0f_9f71_be694b5c88b5.slice/crio-debcf641d2112b798d8da3d81db8296a082178da5ba55ada167f6d898c80fdb4 WatchSource:0}: Error finding container debcf641d2112b798d8da3d81db8296a082178da5ba55ada167f6d898c80fdb4: Status 404 returned error can't find the container with id debcf641d2112b798d8da3d81db8296a082178da5ba55ada167f6d898c80fdb4 Sep 08 04:11:31.453054 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:31.453021 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx" event={"ID":"a342471e-e7cc-4a0f-9f71-be694b5c88b5","Type":"ContainerStarted","Data":"debcf641d2112b798d8da3d81db8296a082178da5ba55ada167f6d898c80fdb4"} Sep 08 04:11:32.460137 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:32.460103 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx" event={"ID":"a342471e-e7cc-4a0f-9f71-be694b5c88b5","Type":"ContainerStarted","Data":"95c670dbb23cb63b3ac912f876bfb78022321bec281d30ae1f9052af35287f6f"} Sep 08 04:11:32.477938 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:32.477888 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx" podStartSLOduration=2.309380433 podStartE2EDuration="2.477873925s" totalImagesPullingTime="168.493492ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:11:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:11:32.475737559 +0000 UTC m=+666.398198318" watchObservedRunningTime="2026-09-08 04:11:32.477873925 +0000 UTC m=+666.400334665" Sep 08 04:11:34.297372 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:34.297337 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:11:41.065217 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:41.065175 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx" Sep 08 04:11:41.065625 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:41.065288 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx" Sep 08 04:11:41.078001 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:41.077973 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx" Sep 08 04:11:41.513662 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:41.513637 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fg2fx" Sep 08 04:11:43.267906 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:43.267849 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:11:48.387549 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:48.387489 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:11:56.142822 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:56.142790 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:11:59.034480 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:11:59.034451 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:12:11.659417 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:12:11.659373 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:12:37.754822 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:12:37.754749 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:12.645450 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:12.645398 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:12.735497 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:12.735462 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:13.719407 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:13.719364 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:13.817996 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:13.817970 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:28.800859 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:28.800815 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:28.920469 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:28.920436 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:29.079335 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.079250 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75"] Sep 08 04:13:29.082442 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.082418 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.084829 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.084805 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ff0f3e-openshift-default-dockercfg-5nrxv\"" Sep 08 04:13:29.094012 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.093985 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75"] Sep 08 04:13:29.099754 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.099730 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48vpp\" (UniqueName: \"kubernetes.io/projected/4517c002-a5f8-4b0f-9002-ec403643ac9f-kube-api-access-48vpp\") pod \"e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.099897 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.099764 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-credential-socket\") pod \"e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.099897 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.099788 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4517c002-a5f8-4b0f-9002-ec403643ac9f-istiod-ca-cert\") pod \"e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.099897 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.099811 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-istio-data\") pod \"e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.099897 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.099833 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4517c002-a5f8-4b0f-9002-ec403643ac9f-istio-podinfo\") pod \"e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.099897 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.099865 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4517c002-a5f8-4b0f-9002-ec403643ac9f-istio-token\") pod \"e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.099897 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.099892 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-istio-envoy\") pod \"e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.100229 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.099920 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-workload-certs\") pod \"e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.100229 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.099947 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-workload-socket\") pod \"e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.109564 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.109532 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:29.200661 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.200632 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48vpp\" (UniqueName: \"kubernetes.io/projected/4517c002-a5f8-4b0f-9002-ec403643ac9f-kube-api-access-48vpp\") pod \"e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.200781 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.200672 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-credential-socket\") pod \"e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.200781 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.200699 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4517c002-a5f8-4b0f-9002-ec403643ac9f-istiod-ca-cert\") pod \"e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.200781 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.200718 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-istio-data\") pod \"e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.200781 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.200734 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4517c002-a5f8-4b0f-9002-ec403643ac9f-istio-podinfo\") pod \"e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.200781 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.200754 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4517c002-a5f8-4b0f-9002-ec403643ac9f-istio-token\") pod \"e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.200781 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.200771 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-istio-envoy\") pod \"e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.201121 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.200800 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-workload-certs\") pod \"e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.201121 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.200848 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-workload-socket\") pod \"e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.201121 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.201038 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-credential-socket\") pod \"e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.201291 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.201174 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-workload-socket\") pod \"e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.201291 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.201243 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-workload-certs\") pod \"e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.201384 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.201343 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-istio-data\") pod \"e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.201626 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.201604 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4517c002-a5f8-4b0f-9002-ec403643ac9f-istiod-ca-cert\") pod \"e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.203442 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.203417 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-istio-envoy\") pod \"e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.203552 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.203427 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4517c002-a5f8-4b0f-9002-ec403643ac9f-istio-podinfo\") pod \"e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.214110 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.214086 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:29.215455 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.215425 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4517c002-a5f8-4b0f-9002-ec403643ac9f-istio-token\") pod \"e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.215879 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.215857 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48vpp\" (UniqueName: \"kubernetes.io/projected/4517c002-a5f8-4b0f-9002-ec403643ac9f-kube-api-access-48vpp\") pod \"e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.394088 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.394012 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:29.672783 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.672757 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75"] Sep 08 04:13:29.674633 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:13:29.674605 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4517c002_a5f8_4b0f_9002_ec403643ac9f.slice/crio-0c91f67f83b97d5716c41b2a4f98fbd09a8c797098ab9725095e9e4d5a513fd1 WatchSource:0}: Error finding container 0c91f67f83b97d5716c41b2a4f98fbd09a8c797098ab9725095e9e4d5a513fd1: Status 404 returned error can't find the container with id 0c91f67f83b97d5716c41b2a4f98fbd09a8c797098ab9725095e9e4d5a513fd1 Sep 08 04:13:29.676647 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.676620 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:13:29.676728 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.676683 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:13:29.676770 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:29.676728 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:13:30.018199 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:30.018158 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" event={"ID":"4517c002-a5f8-4b0f-9002-ec403643ac9f","Type":"ContainerStarted","Data":"e126d8fa775f77354bb1c36c2fc3d0482bd9c259707d5a14233481f386584db1"} Sep 08 04:13:30.018599 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:30.018221 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" event={"ID":"4517c002-a5f8-4b0f-9002-ec403643ac9f","Type":"ContainerStarted","Data":"0c91f67f83b97d5716c41b2a4f98fbd09a8c797098ab9725095e9e4d5a513fd1"} Sep 08 04:13:30.038585 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:30.038535 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" podStartSLOduration=1.038519248 podStartE2EDuration="1.038519248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:13:30.035242902 +0000 UTC m=+783.957703653" watchObservedRunningTime="2026-09-08 04:13:30.038519248 +0000 UTC m=+783.960979984" Sep 08 04:13:30.217575 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:30.217541 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:30.316211 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:30.316124 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:30.394735 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:30.394706 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:30.404522 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:30.400343 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:30.673267 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:30.673163 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:31.023196 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:31.023166 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:31.024415 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:31.024389 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:31.955869 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:31.955831 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:32.054276 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:32.054245 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:33.189187 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:33.189155 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:33.289708 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:33.289663 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:33.678729 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:33.678695 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:33.780198 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:33.780165 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:33.795890 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:33.795861 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c"] Sep 08 04:13:33.799409 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:33.799391 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c" Sep 08 04:13:33.801959 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:33.801941 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ff0f3e-dockercfg-wxs7k\"" Sep 08 04:13:33.802062 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:33.802015 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ff0f3e\"" Sep 08 04:13:33.811495 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:33.811471 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c"] Sep 08 04:13:33.814801 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:33.814779 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7"] Sep 08 04:13:33.817925 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:33.817907 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7" Sep 08 04:13:33.826777 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:33.826752 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7"] Sep 08 04:13:33.840223 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:33.840200 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zl8h\" (UniqueName: \"kubernetes.io/projected/fc7eeac0-8c21-49d3-b9f1-43f850df2cea-kube-api-access-2zl8h\") pod \"payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7\" (UID: \"fc7eeac0-8c21-49d3-b9f1-43f850df2cea\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7" Sep 08 04:13:33.840316 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:33.840238 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7eeac0-8c21-49d3-b9f1-43f850df2cea-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7\" (UID: \"fc7eeac0-8c21-49d3-b9f1-43f850df2cea\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7" Sep 08 04:13:33.840316 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:33.840299 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qsj9\" (UniqueName: \"kubernetes.io/projected/b6973da3-09ec-4aa2-a0dd-e525a8289ca8-kube-api-access-9qsj9\") pod \"payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c\" (UID: \"b6973da3-09ec-4aa2-a0dd-e525a8289ca8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c" Sep 08 04:13:33.840387 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:33.840329 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6973da3-09ec-4aa2-a0dd-e525a8289ca8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c\" (UID: \"b6973da3-09ec-4aa2-a0dd-e525a8289ca8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c" Sep 08 04:13:33.879419 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:33.879382 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:33.941127 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:33.941073 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qsj9\" (UniqueName: \"kubernetes.io/projected/b6973da3-09ec-4aa2-a0dd-e525a8289ca8-kube-api-access-9qsj9\") pod \"payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c\" (UID: \"b6973da3-09ec-4aa2-a0dd-e525a8289ca8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c" Sep 08 04:13:33.941127 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:33.941111 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6973da3-09ec-4aa2-a0dd-e525a8289ca8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c\" (UID: \"b6973da3-09ec-4aa2-a0dd-e525a8289ca8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c" Sep 08 04:13:33.941271 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:33.941155 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zl8h\" (UniqueName: \"kubernetes.io/projected/fc7eeac0-8c21-49d3-b9f1-43f850df2cea-kube-api-access-2zl8h\") pod \"payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7\" (UID: \"fc7eeac0-8c21-49d3-b9f1-43f850df2cea\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7" Sep 08 04:13:33.941271 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:33.941178 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7eeac0-8c21-49d3-b9f1-43f850df2cea-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7\" (UID: \"fc7eeac0-8c21-49d3-b9f1-43f850df2cea\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7" Sep 08 04:13:33.941896 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:33.941876 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7eeac0-8c21-49d3-b9f1-43f850df2cea-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7\" (UID: \"fc7eeac0-8c21-49d3-b9f1-43f850df2cea\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7" Sep 08 04:13:33.941959 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:33.941911 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6973da3-09ec-4aa2-a0dd-e525a8289ca8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c\" (UID: \"b6973da3-09ec-4aa2-a0dd-e525a8289ca8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c" Sep 08 04:13:33.954968 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:33.954946 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zl8h\" (UniqueName: \"kubernetes.io/projected/fc7eeac0-8c21-49d3-b9f1-43f850df2cea-kube-api-access-2zl8h\") pod \"payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7\" (UID: \"fc7eeac0-8c21-49d3-b9f1-43f850df2cea\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7" Sep 08 04:13:33.955085 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:33.955017 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qsj9\" (UniqueName: \"kubernetes.io/projected/b6973da3-09ec-4aa2-a0dd-e525a8289ca8-kube-api-access-9qsj9\") pod \"payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c\" (UID: \"b6973da3-09ec-4aa2-a0dd-e525a8289ca8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c" Sep 08 04:13:34.110426 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:34.110393 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c" Sep 08 04:13:34.129697 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:34.129672 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7" Sep 08 04:13:34.383928 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:34.383902 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c"] Sep 08 04:13:34.385142 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:13:34.385121 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6973da3_09ec_4aa2_a0dd_e525a8289ca8.slice/crio-f957aaa1720c2da9ea909d28a8344afd69871736fb1e00a5ad282db48ac917e9 WatchSource:0}: Error finding container f957aaa1720c2da9ea909d28a8344afd69871736fb1e00a5ad282db48ac917e9: Status 404 returned error can't find the container with id f957aaa1720c2da9ea909d28a8344afd69871736fb1e00a5ad282db48ac917e9 Sep 08 04:13:34.404608 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:34.404583 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7"] Sep 08 04:13:34.405321 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:13:34.405297 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc7eeac0_8c21_49d3_b9f1_43f850df2cea.slice/crio-eef7522e8bd8674fbcf0688437a9ed19018d135caf8da785de51b9815935409d WatchSource:0}: Error finding container eef7522e8bd8674fbcf0688437a9ed19018d135caf8da785de51b9815935409d: Status 404 returned error can't find the container with id eef7522e8bd8674fbcf0688437a9ed19018d135caf8da785de51b9815935409d Sep 08 04:13:35.046452 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:35.046376 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c" event={"ID":"b6973da3-09ec-4aa2-a0dd-e525a8289ca8","Type":"ContainerStarted","Data":"f957aaa1720c2da9ea909d28a8344afd69871736fb1e00a5ad282db48ac917e9"} Sep 08 04:13:35.049993 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:35.049956 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7" event={"ID":"fc7eeac0-8c21-49d3-b9f1-43f850df2cea","Type":"ContainerStarted","Data":"eef7522e8bd8674fbcf0688437a9ed19018d135caf8da785de51b9815935409d"} Sep 08 04:13:36.899686 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:36.899654 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:36.999628 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:36.999591 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:37.062472 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:37.062432 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c" event={"ID":"b6973da3-09ec-4aa2-a0dd-e525a8289ca8","Type":"ContainerStarted","Data":"90e3f0910ea20d2df22fa5da19b54c144c1c8906f17ec3e0d775a31646291598"} Sep 08 04:13:37.062674 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:37.062612 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c" Sep 08 04:13:37.064215 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:37.064194 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7" event={"ID":"fc7eeac0-8c21-49d3-b9f1-43f850df2cea","Type":"ContainerStarted","Data":"e74babdef86ee92638a9c996d22609a7bed987f77ef671f2ea0d1e9d9bbae91c"} Sep 08 04:13:37.064382 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:37.064321 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7" Sep 08 04:13:37.091938 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:37.091891 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c" podStartSLOduration=2.361668516 podStartE2EDuration="4.091876528s" totalImagesPullingTime="1.730208012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:13:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:13:37.090853971 +0000 UTC m=+791.013314730" watchObservedRunningTime="2026-09-08 04:13:37.091876528 +0000 UTC m=+791.014337267" Sep 08 04:13:37.117709 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:37.117667 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7" podStartSLOduration=2.397212459 podStartE2EDuration="4.11765527s" totalImagesPullingTime="1.720442811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:13:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:13:37.115603848 +0000 UTC m=+791.038064609" watchObservedRunningTime="2026-09-08 04:13:37.11765527 +0000 UTC m=+791.040116009" Sep 08 04:13:37.142543 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:37.142494 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:37.528686 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:37.528647 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:37.956536 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:37.956431 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:38.902305 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:38.902223 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:39.166391 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.166309 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp"] Sep 08 04:13:39.170023 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.169993 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.172481 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.172458 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-f0271e-openshift-default-dockercfg-qhs4w\"" Sep 08 04:13:39.183014 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.182986 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp"] Sep 08 04:13:39.198594 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.198568 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:39.294818 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.294787 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istio-token\") pod \"e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.294992 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.294845 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-workload-certs\") pod \"e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.294992 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.294940 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istio-podinfo\") pod \"e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.295074 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.294988 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istiod-ca-cert\") pod \"e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.295074 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.295032 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-workload-socket\") pod \"e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.295074 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.295065 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istio-envoy\") pod \"e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.295173 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.295135 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-credential-socket\") pod \"e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.295212 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.295181 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istio-data\") pod \"e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.295212 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.295198 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kms6f\" (UniqueName: \"kubernetes.io/projected/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-kube-api-access-kms6f\") pod \"e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.351976 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.351943 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:39.396223 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.396186 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-workload-socket\") pod \"e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.396910 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.396889 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istio-envoy\") pod \"e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.397140 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.397123 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-credential-socket\") pod \"e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.397608 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.397589 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istio-data\") pod \"e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.398041 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.398025 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kms6f\" (UniqueName: \"kubernetes.io/projected/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-kube-api-access-kms6f\") pod \"e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.398619 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.398598 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istio-token\") pod \"e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.399102 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.399082 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-workload-certs\") pod \"e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.399732 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.399704 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istio-podinfo\") pod \"e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.399918 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.399901 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istiod-ca-cert\") pod \"e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.400749 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.399545 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-workload-certs\") pod \"e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.400886 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.397527 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-credential-socket\") pod \"e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.401046 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.399823 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istio-envoy\") pod \"e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.401134 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.396839 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-workload-socket\") pod \"e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.401228 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.397970 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istio-data\") pod \"e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.401327 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.400660 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istiod-ca-cert\") pod \"e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.404062 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.404020 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istio-podinfo\") pod \"e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.409163 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.409134 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kms6f\" (UniqueName: \"kubernetes.io/projected/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-kube-api-access-kms6f\") pod \"e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.414959 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.414928 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istio-token\") pod \"e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.483543 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.483453 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:39.779573 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.779438 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp"] Sep 08 04:13:39.782039 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:13:39.782014 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb216dc0_7136_4cb5_b513_b8cf7ab9252c.slice/crio-c5c8f79a19d59bd987b002da54d28c09c3dcfa059872309204d4d79845970ae3 WatchSource:0}: Error finding container c5c8f79a19d59bd987b002da54d28c09c3dcfa059872309204d4d79845970ae3: Status 404 returned error can't find the container with id c5c8f79a19d59bd987b002da54d28c09c3dcfa059872309204d4d79845970ae3 Sep 08 04:13:39.784660 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.784631 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:13:39.784762 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.784705 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:13:39.784762 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:39.784751 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:13:40.081958 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:40.081782 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" event={"ID":"fb216dc0-7136-4cb5-b513-b8cf7ab9252c","Type":"ContainerStarted","Data":"a14b5019a938da481136fd6338eee6167e23ff01fad56d758b9137081107a6ce"} Sep 08 04:13:40.081958 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:40.081896 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" event={"ID":"fb216dc0-7136-4cb5-b513-b8cf7ab9252c","Type":"ContainerStarted","Data":"c5c8f79a19d59bd987b002da54d28c09c3dcfa059872309204d4d79845970ae3"} Sep 08 04:13:40.123247 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:40.123203 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:40.223766 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:40.223727 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:40.484656 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:40.484564 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:40.490044 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:40.490020 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:40.539050 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:40.538994 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" podStartSLOduration=1.53897482 podStartE2EDuration="1.53897482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:13:40.107591478 +0000 UTC m=+794.030052233" watchObservedRunningTime="2026-09-08 04:13:40.53897482 +0000 UTC m=+794.461435560" Sep 08 04:13:41.088187 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:41.088153 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:41.089203 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:41.089183 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:13:41.332617 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:41.332583 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:41.402590 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:41.400245 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:41.468414 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:41.468373 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:41.773012 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:41.772980 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:41.906314 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:41.901353 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:41.995559 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:41.995519 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:42.075273 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:42.075173 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:42.144039 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:42.144010 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:43.209718 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:43.209279 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:43.301175 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:43.301133 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:43.857460 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:43.857424 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:43.962668 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:43.962634 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4"] Sep 08 04:13:43.963462 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:43.963440 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:43.966065 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:43.966050 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4" Sep 08 04:13:43.968758 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:43.968736 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-f0271e\"" Sep 08 04:13:43.969002 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:43.968985 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-f0271e-dockercfg-f62jw\"" Sep 08 04:13:43.976251 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:43.976226 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4"] Sep 08 04:13:43.995545 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:43.993071 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f0271e-8558c55579-6jvbf"] Sep 08 04:13:44.023529 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:44.023128 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-f0271e-8558c55579-6jvbf" Sep 08 04:13:44.023529 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:44.023403 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f0271e-8558c55579-6jvbf"] Sep 08 04:13:44.043926 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:44.043897 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:44.053640 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:44.053616 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgl88\" (UniqueName: \"kubernetes.io/projected/247f46a3-f13a-473a-9100-72fc5f426006-kube-api-access-hgl88\") pod \"payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4\" (UID: \"247f46a3-f13a-473a-9100-72fc5f426006\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4" Sep 08 04:13:44.053744 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:44.053647 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247f46a3-f13a-473a-9100-72fc5f426006-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4\" (UID: \"247f46a3-f13a-473a-9100-72fc5f426006\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4" Sep 08 04:13:44.154613 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:44.154523 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0384d0ca-874e-495a-9764-70043ed9bb83-plugins-config-volume\") pod \"payload-processing-e2e-autores-f0271e-8558c55579-6jvbf\" (UID: \"0384d0ca-874e-495a-9764-70043ed9bb83\") " pod="openshift-ingress/payload-processing-e2e-autores-f0271e-8558c55579-6jvbf" Sep 08 04:13:44.154790 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:44.154759 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chnwl\" (UniqueName: \"kubernetes.io/projected/0384d0ca-874e-495a-9764-70043ed9bb83-kube-api-access-chnwl\") pod \"payload-processing-e2e-autores-f0271e-8558c55579-6jvbf\" (UID: \"0384d0ca-874e-495a-9764-70043ed9bb83\") " pod="openshift-ingress/payload-processing-e2e-autores-f0271e-8558c55579-6jvbf" Sep 08 04:13:44.154866 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:44.154845 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgl88\" (UniqueName: \"kubernetes.io/projected/247f46a3-f13a-473a-9100-72fc5f426006-kube-api-access-hgl88\") pod \"payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4\" (UID: \"247f46a3-f13a-473a-9100-72fc5f426006\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4" Sep 08 04:13:44.154922 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:44.154878 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247f46a3-f13a-473a-9100-72fc5f426006-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4\" (UID: \"247f46a3-f13a-473a-9100-72fc5f426006\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4" Sep 08 04:13:44.155872 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:44.155845 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247f46a3-f13a-473a-9100-72fc5f426006-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4\" (UID: \"247f46a3-f13a-473a-9100-72fc5f426006\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4" Sep 08 04:13:44.162935 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:44.162903 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgl88\" (UniqueName: \"kubernetes.io/projected/247f46a3-f13a-473a-9100-72fc5f426006-kube-api-access-hgl88\") pod \"payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4\" (UID: \"247f46a3-f13a-473a-9100-72fc5f426006\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4" Sep 08 04:13:44.255658 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:44.255622 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0384d0ca-874e-495a-9764-70043ed9bb83-plugins-config-volume\") pod \"payload-processing-e2e-autores-f0271e-8558c55579-6jvbf\" (UID: \"0384d0ca-874e-495a-9764-70043ed9bb83\") " pod="openshift-ingress/payload-processing-e2e-autores-f0271e-8558c55579-6jvbf" Sep 08 04:13:44.256008 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:44.255672 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chnwl\" (UniqueName: \"kubernetes.io/projected/0384d0ca-874e-495a-9764-70043ed9bb83-kube-api-access-chnwl\") pod \"payload-processing-e2e-autores-f0271e-8558c55579-6jvbf\" (UID: \"0384d0ca-874e-495a-9764-70043ed9bb83\") " pod="openshift-ingress/payload-processing-e2e-autores-f0271e-8558c55579-6jvbf" Sep 08 04:13:44.256307 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:44.256285 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0384d0ca-874e-495a-9764-70043ed9bb83-plugins-config-volume\") pod \"payload-processing-e2e-autores-f0271e-8558c55579-6jvbf\" (UID: \"0384d0ca-874e-495a-9764-70043ed9bb83\") " pod="openshift-ingress/payload-processing-e2e-autores-f0271e-8558c55579-6jvbf" Sep 08 04:13:44.264109 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:44.264084 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chnwl\" (UniqueName: \"kubernetes.io/projected/0384d0ca-874e-495a-9764-70043ed9bb83-kube-api-access-chnwl\") pod \"payload-processing-e2e-autores-f0271e-8558c55579-6jvbf\" (UID: \"0384d0ca-874e-495a-9764-70043ed9bb83\") " pod="openshift-ingress/payload-processing-e2e-autores-f0271e-8558c55579-6jvbf" Sep 08 04:13:44.277926 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:44.277902 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4" Sep 08 04:13:44.335297 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:44.335249 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-f0271e-8558c55579-6jvbf" Sep 08 04:13:44.572564 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:44.572536 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4"] Sep 08 04:13:44.573594 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:13:44.573564 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247f46a3_f13a_473a_9100_72fc5f426006.slice/crio-2288216a6d1de1cb24b17970684e3a1966e614477bea5559e751426d86279805 WatchSource:0}: Error finding container 2288216a6d1de1cb24b17970684e3a1966e614477bea5559e751426d86279805: Status 404 returned error can't find the container with id 2288216a6d1de1cb24b17970684e3a1966e614477bea5559e751426d86279805 Sep 08 04:13:44.662395 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:44.662369 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f0271e-8558c55579-6jvbf"] Sep 08 04:13:44.664193 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:13:44.664167 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0384d0ca_874e_495a_9764_70043ed9bb83.slice/crio-74eb1b8e610989501588363fd5c121d57e5b2c27cacd8db250b20811671ccd41 WatchSource:0}: Error finding container 74eb1b8e610989501588363fd5c121d57e5b2c27cacd8db250b20811671ccd41: Status 404 returned error can't find the container with id 74eb1b8e610989501588363fd5c121d57e5b2c27cacd8db250b20811671ccd41 Sep 08 04:13:45.133576 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:45.133529 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-f0271e-8558c55579-6jvbf" event={"ID":"0384d0ca-874e-495a-9764-70043ed9bb83","Type":"ContainerStarted","Data":"54d883776d9612322fb2a071e2bb7fff16ea31c0a25abecca86dc47b69e66f89"} Sep 08 04:13:45.133763 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:45.133601 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-f0271e-8558c55579-6jvbf" Sep 08 04:13:45.133763 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:45.133628 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-f0271e-8558c55579-6jvbf" event={"ID":"0384d0ca-874e-495a-9764-70043ed9bb83","Type":"ContainerStarted","Data":"74eb1b8e610989501588363fd5c121d57e5b2c27cacd8db250b20811671ccd41"} Sep 08 04:13:45.135316 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:45.135289 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4" event={"ID":"247f46a3-f13a-473a-9100-72fc5f426006","Type":"ContainerStarted","Data":"97b5c2c7bd740d67a242cceb79c5f10db13cdd437b5441c00dc0362837e7fb4d"} Sep 08 04:13:45.135457 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:45.135340 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4" event={"ID":"247f46a3-f13a-473a-9100-72fc5f426006","Type":"ContainerStarted","Data":"2288216a6d1de1cb24b17970684e3a1966e614477bea5559e751426d86279805"} Sep 08 04:13:45.135457 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:45.135367 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4" Sep 08 04:13:45.152573 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:45.152492 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-f0271e-8558c55579-6jvbf" podStartSLOduration=2.152466812 podStartE2EDuration="2.152466812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:13:45.149736649 +0000 UTC m=+799.072197411" watchObservedRunningTime="2026-09-08 04:13:45.152466812 +0000 UTC m=+799.074927550" Sep 08 04:13:45.163893 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:45.163784 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4" podStartSLOduration=2.163767626 podStartE2EDuration="2.163767626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:13:45.163522481 +0000 UTC m=+799.085983241" watchObservedRunningTime="2026-09-08 04:13:45.163767626 +0000 UTC m=+799.086228346" Sep 08 04:13:47.065659 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:47.065626 2574 patch_prober.go:28] interesting pod/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 08 04:13:47.066038 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:47.065697 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" podUID="4517c002-a5f8-4b0f-9002-ec403643ac9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 08 04:13:47.779770 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:47.779738 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:47.921877 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:47.921840 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:48.025405 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:48.025322 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:48.071598 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:48.071569 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c" Sep 08 04:13:48.071896 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:48.071718 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7" Sep 08 04:13:48.090707 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:48.090685 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:48.152275 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:48.152249 2574 generic.go:356] "Generic (PLEG): container finished" podID="4517c002-a5f8-4b0f-9002-ec403643ac9f" containerID="e126d8fa775f77354bb1c36c2fc3d0482bd9c259707d5a14233481f386584db1" exitCode=0 Sep 08 04:13:48.152402 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:48.152317 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" event={"ID":"4517c002-a5f8-4b0f-9002-ec403643ac9f","Type":"ContainerDied","Data":"e126d8fa775f77354bb1c36c2fc3d0482bd9c259707d5a14233481f386584db1"} Sep 08 04:13:48.152703 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:48.152688 2574 scope.go:117] "RemoveContainer" containerID="e126d8fa775f77354bb1c36c2fc3d0482bd9c259707d5a14233481f386584db1" Sep 08 04:13:48.153762 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:48.153732 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:13:48.153864 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:48.153837 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:13:48.153927 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:48.153888 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:13:49.159841 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:49.159764 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" event={"ID":"4517c002-a5f8-4b0f-9002-ec403643ac9f","Type":"ContainerStarted","Data":"57ce77fc327c64d444745bbafbc897a5e2d282cd19927ab78b03a314cc8fb593"} Sep 08 04:13:49.395218 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:49.395176 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:49.395423 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:49.395309 2574 patch_prober.go:28] interesting pod/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Sep 08 04:13:49.395423 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:49.395354 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" podUID="4517c002-a5f8-4b0f-9002-ec403643ac9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Sep 08 04:13:49.803418 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:49.803324 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:49.888156 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:49.888121 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:50.054182 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:50.054103 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:50.395316 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:50.394823 2574 patch_prober.go:28] interesting pod/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Sep 08 04:13:50.395316 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:50.394884 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" podUID="4517c002-a5f8-4b0f-9002-ec403643ac9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Sep 08 04:13:50.791469 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:50.791433 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:51.394794 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:51.394754 2574 patch_prober.go:28] interesting pod/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Sep 08 04:13:51.394986 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:51.394820 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" podUID="4517c002-a5f8-4b0f-9002-ec403643ac9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Sep 08 04:13:52.369597 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:52.369563 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:52.394443 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:52.394402 2574 patch_prober.go:28] interesting pod/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Sep 08 04:13:52.394611 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:52.394478 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" podUID="4517c002-a5f8-4b0f-9002-ec403643ac9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Sep 08 04:13:53.398922 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:53.398877 2574 patch_prober.go:28] interesting pod/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Sep 08 04:13:53.399400 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:53.398956 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" podUID="4517c002-a5f8-4b0f-9002-ec403643ac9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Sep 08 04:13:54.397558 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:54.397530 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:54.397838 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:54.397816 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:54.398437 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:54.398417 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:13:55.282097 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:55.282062 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:56.142436 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:56.142394 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4" Sep 08 04:13:56.142623 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:56.142555 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-f0271e-8558c55579-6jvbf" Sep 08 04:13:56.528552 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:56.528490 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:56.629679 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:56.629647 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:56.729252 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:56.729210 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:56.878887 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:56.878810 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:56.978870 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:56.978838 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:57.251381 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:57.251347 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:57.564315 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:57.564231 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:13:58.311741 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:13:58.311706 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:00.801360 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:00.801324 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:00.873859 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:00.873829 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:01.975472 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:01.975441 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:02.075941 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:02.075858 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:02.458871 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:02.458796 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:02.559790 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:02.559745 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:02.897442 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:02.897413 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:03.596237 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:03.596162 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:03.696570 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:03.696540 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:03.803687 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:03.803655 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:04.468625 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:04.468590 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:05.259116 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:05.259087 2574 generic.go:356] "Generic (PLEG): container finished" podID="fb216dc0-7136-4cb5-b513-b8cf7ab9252c" containerID="a14b5019a938da481136fd6338eee6167e23ff01fad56d758b9137081107a6ce" exitCode=0 Sep 08 04:14:05.259463 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:05.259159 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" event={"ID":"fb216dc0-7136-4cb5-b513-b8cf7ab9252c","Type":"ContainerDied","Data":"a14b5019a938da481136fd6338eee6167e23ff01fad56d758b9137081107a6ce"} Sep 08 04:14:05.259525 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:05.259483 2574 scope.go:117] "RemoveContainer" containerID="a14b5019a938da481136fd6338eee6167e23ff01fad56d758b9137081107a6ce" Sep 08 04:14:05.260582 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:05.260556 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:14:05.260654 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:05.260619 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:14:05.260654 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:05.260651 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:14:05.481408 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:05.481363 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:05.601879 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:05.601791 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d710ebd-bp9tz"] Sep 08 04:14:05.605571 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:05.605541 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d710ebd-bp9tz" Sep 08 04:14:05.611699 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:05.608182 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7r29w\"" Sep 08 04:14:05.613760 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:05.612249 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 04:14:05.615302 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:05.615213 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d710ebd-bp9tz"] Sep 08 04:14:05.755928 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:05.755899 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26a7857-75f8-4007-852d-68758f763f86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8d710ebd-bp9tz\" (UID: \"c26a7857-75f8-4007-852d-68758f763f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d710ebd-bp9tz" Sep 08 04:14:05.857236 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:05.857158 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26a7857-75f8-4007-852d-68758f763f86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8d710ebd-bp9tz\" (UID: \"c26a7857-75f8-4007-852d-68758f763f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d710ebd-bp9tz" Sep 08 04:14:05.857868 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:05.857837 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26a7857-75f8-4007-852d-68758f763f86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8d710ebd-bp9tz\" (UID: \"c26a7857-75f8-4007-852d-68758f763f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d710ebd-bp9tz" Sep 08 04:14:05.926473 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:05.926443 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d710ebd-bp9tz" Sep 08 04:14:06.210011 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:06.209968 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d710ebd-bp9tz"] Sep 08 04:14:06.212002 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:14:06.211975 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26a7857_75f8_4007_852d_68758f763f86.slice/crio-4d9826036537d7665975e54c3040c31e190009afa196f11852eb19c75460e2c9 WatchSource:0}: Error finding container 4d9826036537d7665975e54c3040c31e190009afa196f11852eb19c75460e2c9: Status 404 returned error can't find the container with id 4d9826036537d7665975e54c3040c31e190009afa196f11852eb19c75460e2c9 Sep 08 04:14:06.267334 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:06.267299 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" event={"ID":"fb216dc0-7136-4cb5-b513-b8cf7ab9252c","Type":"ContainerStarted","Data":"dca3efd53ea5ed84c88ca192ded87e312d543931c9be757a1e7f423daa43df6a"} Sep 08 04:14:06.268590 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:06.268558 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d710ebd-bp9tz" event={"ID":"c26a7857-75f8-4007-852d-68758f763f86","Type":"ContainerStarted","Data":"4d9826036537d7665975e54c3040c31e190009afa196f11852eb19c75460e2c9"} Sep 08 04:14:06.326205 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:06.326173 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:06.484645 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:06.484556 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:14:06.484848 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:06.484824 2574 patch_prober.go:28] interesting pod/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 08 04:14:06.484930 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:06.484883 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" podUID="fb216dc0-7136-4cb5-b513-b8cf7ab9252c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 08 04:14:06.952613 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:06.952575 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:07.243708 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:07.243624 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:07.484415 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:07.484379 2574 patch_prober.go:28] interesting pod/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 08 04:14:07.484861 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:07.484449 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" podUID="fb216dc0-7136-4cb5-b513-b8cf7ab9252c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 08 04:14:08.484312 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:08.484273 2574 patch_prober.go:28] interesting pod/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 08 04:14:08.484512 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:08.484333 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" podUID="fb216dc0-7136-4cb5-b513-b8cf7ab9252c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 08 04:14:09.484431 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:09.484396 2574 patch_prober.go:28] interesting pod/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 08 04:14:09.484599 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:09.484446 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" podUID="fb216dc0-7136-4cb5-b513-b8cf7ab9252c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 08 04:14:10.295975 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:10.295937 2574 generic.go:356] "Generic (PLEG): container finished" podID="c26a7857-75f8-4007-852d-68758f763f86" containerID="c3673fd3ccfd0ba4708a95eb23019e4c0ddf493ba6e0f509c5854b18c237b843" exitCode=0 Sep 08 04:14:10.296144 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:10.296022 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d710ebd-bp9tz" event={"ID":"c26a7857-75f8-4007-852d-68758f763f86","Type":"ContainerDied","Data":"c3673fd3ccfd0ba4708a95eb23019e4c0ddf493ba6e0f509c5854b18c237b843"} Sep 08 04:14:10.484396 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:10.484365 2574 patch_prober.go:28] interesting pod/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 08 04:14:10.484578 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:10.484417 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" podUID="fb216dc0-7136-4cb5-b513-b8cf7ab9252c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 08 04:14:11.088343 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:11.088312 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:14:11.469828 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:11.469802 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d710ebd-bp9tz" Sep 08 04:14:11.484208 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:11.484179 2574 patch_prober.go:28] interesting pod/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 08 04:14:11.484358 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:11.484240 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" podUID="fb216dc0-7136-4cb5-b513-b8cf7ab9252c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 08 04:14:11.614580 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:11.611799 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26a7857-75f8-4007-852d-68758f763f86-maas-api-service-ca\") pod \"c26a7857-75f8-4007-852d-68758f763f86\" (UID: \"c26a7857-75f8-4007-852d-68758f763f86\") " Sep 08 04:14:11.614580 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:11.612311 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c26a7857-75f8-4007-852d-68758f763f86-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c26a7857-75f8-4007-852d-68758f763f86" (UID: "c26a7857-75f8-4007-852d-68758f763f86"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:14:11.713190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:11.712739 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26a7857-75f8-4007-852d-68758f763f86-maas-api-service-ca\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:14:12.312477 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:12.312445 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d710ebd-bp9tz" Sep 08 04:14:12.312477 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:12.312464 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d710ebd-bp9tz" event={"ID":"c26a7857-75f8-4007-852d-68758f763f86","Type":"ContainerDied","Data":"4d9826036537d7665975e54c3040c31e190009afa196f11852eb19c75460e2c9"} Sep 08 04:14:12.312906 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:12.312556 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d9826036537d7665975e54c3040c31e190009afa196f11852eb19c75460e2c9" Sep 08 04:14:12.488369 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:12.488346 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:14:12.489315 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:12.489294 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:14:20.352040 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:20.351996 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:21.473662 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:21.473608 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:22.086292 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:22.086260 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:22.163364 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:22.163328 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:22.263620 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:22.263584 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:22.309037 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:22.309002 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d710ebd-bp9tz"] Sep 08 04:14:22.311872 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:22.311849 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d710ebd-bp9tz"] Sep 08 04:14:22.318104 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:22.318084 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:22.442646 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:22.442566 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:22.567953 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:22.567919 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c26a7857-75f8-4007-852d-68758f763f86" path="/var/lib/kubelet/pods/c26a7857-75f8-4007-852d-68758f763f86/volumes" Sep 08 04:14:22.887519 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:22.887480 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:22.984231 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:22.984206 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:23.175602 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:23.175500 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:23.253067 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:23.253027 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:23.414909 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:23.414868 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:23.711932 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:23.711902 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:25.746061 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:25.746026 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:26.454684 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:26.454656 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:26.558760 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:26.558728 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:26.720396 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:26.720297 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:26.969887 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:26.969853 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:27.244063 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.244035 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:27.346323 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.346298 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:27.354420 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.354396 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn"] Sep 08 04:14:27.354792 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.354779 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c26a7857-75f8-4007-852d-68758f763f86" containerName="revoke-keys" Sep 08 04:14:27.354848 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.354793 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c26a7857-75f8-4007-852d-68758f763f86" containerName="revoke-keys" Sep 08 04:14:27.354887 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.354855 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c26a7857-75f8-4007-852d-68758f763f86" containerName="revoke-keys" Sep 08 04:14:27.358019 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.358003 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn" Sep 08 04:14:27.361021 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.360999 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-4cc32bbe-dockercfg-tlmfw\"" Sep 08 04:14:27.361021 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.361013 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-4cc32bbe\"" Sep 08 04:14:27.366733 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.366711 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn"] Sep 08 04:14:27.370639 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.370619 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7"] Sep 08 04:14:27.373748 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.373732 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7" Sep 08 04:14:27.383622 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.383600 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7"] Sep 08 04:14:27.452836 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.452794 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15332324-579d-429d-97c0-6fbe37fea804-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7\" (UID: \"15332324-579d-429d-97c0-6fbe37fea804\") " pod="openshift-ingress/payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7" Sep 08 04:14:27.453020 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.452927 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg4d6\" (UniqueName: \"kubernetes.io/projected/15332324-579d-429d-97c0-6fbe37fea804-kube-api-access-qg4d6\") pod \"payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7\" (UID: \"15332324-579d-429d-97c0-6fbe37fea804\") " pod="openshift-ingress/payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7" Sep 08 04:14:27.453020 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.452991 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7057246-1d51-4e96-8f0f-487a16dfadd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn\" (UID: \"c7057246-1d51-4e96-8f0f-487a16dfadd2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn" Sep 08 04:14:27.453128 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.453042 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqr4q\" (UniqueName: \"kubernetes.io/projected/c7057246-1d51-4e96-8f0f-487a16dfadd2-kube-api-access-lqr4q\") pod \"payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn\" (UID: \"c7057246-1d51-4e96-8f0f-487a16dfadd2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn" Sep 08 04:14:27.553920 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.553889 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqr4q\" (UniqueName: \"kubernetes.io/projected/c7057246-1d51-4e96-8f0f-487a16dfadd2-kube-api-access-lqr4q\") pod \"payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn\" (UID: \"c7057246-1d51-4e96-8f0f-487a16dfadd2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn" Sep 08 04:14:27.554090 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.553946 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15332324-579d-429d-97c0-6fbe37fea804-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7\" (UID: \"15332324-579d-429d-97c0-6fbe37fea804\") " pod="openshift-ingress/payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7" Sep 08 04:14:27.554090 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.554003 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg4d6\" (UniqueName: \"kubernetes.io/projected/15332324-579d-429d-97c0-6fbe37fea804-kube-api-access-qg4d6\") pod \"payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7\" (UID: \"15332324-579d-429d-97c0-6fbe37fea804\") " pod="openshift-ingress/payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7" Sep 08 04:14:27.554090 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.554020 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7057246-1d51-4e96-8f0f-487a16dfadd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn\" (UID: \"c7057246-1d51-4e96-8f0f-487a16dfadd2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn" Sep 08 04:14:27.554666 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.554645 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15332324-579d-429d-97c0-6fbe37fea804-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7\" (UID: \"15332324-579d-429d-97c0-6fbe37fea804\") " pod="openshift-ingress/payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7" Sep 08 04:14:27.554750 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.554733 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7057246-1d51-4e96-8f0f-487a16dfadd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn\" (UID: \"c7057246-1d51-4e96-8f0f-487a16dfadd2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn" Sep 08 04:14:27.561998 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.561969 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg4d6\" (UniqueName: \"kubernetes.io/projected/15332324-579d-429d-97c0-6fbe37fea804-kube-api-access-qg4d6\") pod \"payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7\" (UID: \"15332324-579d-429d-97c0-6fbe37fea804\") " pod="openshift-ingress/payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7" Sep 08 04:14:27.562101 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.562006 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqr4q\" (UniqueName: \"kubernetes.io/projected/c7057246-1d51-4e96-8f0f-487a16dfadd2-kube-api-access-lqr4q\") pod \"payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn\" (UID: \"c7057246-1d51-4e96-8f0f-487a16dfadd2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn" Sep 08 04:14:27.671025 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.670979 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn" Sep 08 04:14:27.684200 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.684175 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7" Sep 08 04:14:27.946450 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.946422 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn"] Sep 08 04:14:27.947602 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:14:27.947577 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7057246_1d51_4e96_8f0f_487a16dfadd2.slice/crio-969a0dcb9a7c8e8d14b5b460c03cda45301002241761e3c98fb2ad214f884372 WatchSource:0}: Error finding container 969a0dcb9a7c8e8d14b5b460c03cda45301002241761e3c98fb2ad214f884372: Status 404 returned error can't find the container with id 969a0dcb9a7c8e8d14b5b460c03cda45301002241761e3c98fb2ad214f884372 Sep 08 04:14:27.977680 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:27.977660 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7"] Sep 08 04:14:27.983559 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:14:27.983527 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15332324_579d_429d_97c0_6fbe37fea804.slice/crio-007a1562d96ca5f675ef8295f0746c1fe1bda8cf8e701c2d0381c2a83ffb3842 WatchSource:0}: Error finding container 007a1562d96ca5f675ef8295f0746c1fe1bda8cf8e701c2d0381c2a83ffb3842: Status 404 returned error can't find the container with id 007a1562d96ca5f675ef8295f0746c1fe1bda8cf8e701c2d0381c2a83ffb3842 Sep 08 04:14:28.408071 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:28.408035 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7" event={"ID":"15332324-579d-429d-97c0-6fbe37fea804","Type":"ContainerStarted","Data":"926f047b28d3ac78ad23765e8a79b70bea60a9b09a06e6d19a46453bd3a2975c"} Sep 08 04:14:28.408268 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:28.408129 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7" event={"ID":"15332324-579d-429d-97c0-6fbe37fea804","Type":"ContainerStarted","Data":"007a1562d96ca5f675ef8295f0746c1fe1bda8cf8e701c2d0381c2a83ffb3842"} Sep 08 04:14:28.408268 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:28.408162 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7" Sep 08 04:14:28.409822 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:28.409787 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn" event={"ID":"c7057246-1d51-4e96-8f0f-487a16dfadd2","Type":"ContainerStarted","Data":"ee9fd64722f9125a586d2656890c8c110589127acf1aefbf97e17dd5a7240d43"} Sep 08 04:14:28.409958 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:28.409841 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn" event={"ID":"c7057246-1d51-4e96-8f0f-487a16dfadd2","Type":"ContainerStarted","Data":"969a0dcb9a7c8e8d14b5b460c03cda45301002241761e3c98fb2ad214f884372"} Sep 08 04:14:28.409958 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:28.409862 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn" Sep 08 04:14:28.425232 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:28.425178 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7" podStartSLOduration=1.425161853 podStartE2EDuration="1.425161853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:14:28.424548724 +0000 UTC m=+842.347009490" watchObservedRunningTime="2026-09-08 04:14:28.425161853 +0000 UTC m=+842.347622595" Sep 08 04:14:28.440716 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:28.440666 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn" podStartSLOduration=1.4406492100000001 podStartE2EDuration="1.44064921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:14:28.439763552 +0000 UTC m=+842.362224327" watchObservedRunningTime="2026-09-08 04:14:28.44064921 +0000 UTC m=+842.363109954" Sep 08 04:14:28.786393 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:28.786365 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:39.417820 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:39.417770 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7" Sep 08 04:14:39.418313 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:39.418008 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn" Sep 08 04:14:49.616608 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:49.616565 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:51.406765 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:51.406730 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cc32bbe-gzhh9"] Sep 08 04:14:51.413914 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:51.413886 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cc32bbe-gzhh9" Sep 08 04:14:51.416332 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:51.416303 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 04:14:51.416332 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:51.416309 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7r29w\"" Sep 08 04:14:51.416680 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:51.416652 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cc32bbe-gzhh9"] Sep 08 04:14:51.547392 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:51.547363 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0aa30b0-d047-4617-a435-5229ab88e850-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4cc32bbe-gzhh9\" (UID: \"e0aa30b0-d047-4617-a435-5229ab88e850\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cc32bbe-gzhh9" Sep 08 04:14:51.648772 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:51.648748 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0aa30b0-d047-4617-a435-5229ab88e850-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4cc32bbe-gzhh9\" (UID: \"e0aa30b0-d047-4617-a435-5229ab88e850\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cc32bbe-gzhh9" Sep 08 04:14:51.649414 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:51.649392 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0aa30b0-d047-4617-a435-5229ab88e850-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4cc32bbe-gzhh9\" (UID: \"e0aa30b0-d047-4617-a435-5229ab88e850\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cc32bbe-gzhh9" Sep 08 04:14:51.671102 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:51.671047 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:51.728193 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:51.728173 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cc32bbe-gzhh9" Sep 08 04:14:51.756378 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:51.756356 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:52.004593 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:52.004564 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cc32bbe-gzhh9"] Sep 08 04:14:52.006181 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:14:52.006159 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0aa30b0_d047_4617_a435_5229ab88e850.slice/crio-d32fd6f5c5aa402729a544a88c7080d030d89a1df368df6ba8fb1270f066c02a WatchSource:0}: Error finding container d32fd6f5c5aa402729a544a88c7080d030d89a1df368df6ba8fb1270f066c02a: Status 404 returned error can't find the container with id d32fd6f5c5aa402729a544a88c7080d030d89a1df368df6ba8fb1270f066c02a Sep 08 04:14:52.555803 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:52.555772 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cc32bbe-gzhh9" event={"ID":"e0aa30b0-d047-4617-a435-5229ab88e850","Type":"ContainerStarted","Data":"48b54313e852d54be3d45d027a1ecc79cc6bd4d15eb3eace483cf7ec3a0edd65"} Sep 08 04:14:52.556233 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:52.555830 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cc32bbe-gzhh9" event={"ID":"e0aa30b0-d047-4617-a435-5229ab88e850","Type":"ContainerStarted","Data":"d32fd6f5c5aa402729a544a88c7080d030d89a1df368df6ba8fb1270f066c02a"} Sep 08 04:14:52.573052 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:52.573007 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cc32bbe-gzhh9" podStartSLOduration=1.572994235 podStartE2EDuration="1.572994235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:14:52.572731378 +0000 UTC m=+866.495192137" watchObservedRunningTime="2026-09-08 04:14:52.572994235 +0000 UTC m=+866.495454974" Sep 08 04:14:53.571227 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:53.571192 2574 generic.go:356] "Generic (PLEG): container finished" podID="e0aa30b0-d047-4617-a435-5229ab88e850" containerID="48b54313e852d54be3d45d027a1ecc79cc6bd4d15eb3eace483cf7ec3a0edd65" exitCode=0 Sep 08 04:14:53.571617 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:53.571272 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cc32bbe-gzhh9" event={"ID":"e0aa30b0-d047-4617-a435-5229ab88e850","Type":"ContainerDied","Data":"48b54313e852d54be3d45d027a1ecc79cc6bd4d15eb3eace483cf7ec3a0edd65"} Sep 08 04:14:54.710742 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:54.710712 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cc32bbe-gzhh9" Sep 08 04:14:54.777611 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:54.777587 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0aa30b0-d047-4617-a435-5229ab88e850-maas-api-service-ca\") pod \"e0aa30b0-d047-4617-a435-5229ab88e850\" (UID: \"e0aa30b0-d047-4617-a435-5229ab88e850\") " Sep 08 04:14:54.777897 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:54.777876 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0aa30b0-d047-4617-a435-5229ab88e850-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e0aa30b0-d047-4617-a435-5229ab88e850" (UID: "e0aa30b0-d047-4617-a435-5229ab88e850"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:14:54.878245 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:54.878182 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0aa30b0-d047-4617-a435-5229ab88e850-maas-api-service-ca\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:14:55.583069 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:55.583037 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cc32bbe-gzhh9" event={"ID":"e0aa30b0-d047-4617-a435-5229ab88e850","Type":"ContainerDied","Data":"d32fd6f5c5aa402729a544a88c7080d030d89a1df368df6ba8fb1270f066c02a"} Sep 08 04:14:55.583218 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:55.583083 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d32fd6f5c5aa402729a544a88c7080d030d89a1df368df6ba8fb1270f066c02a" Sep 08 04:14:55.583218 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:55.583049 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cc32bbe-gzhh9" Sep 08 04:14:57.919310 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:57.919280 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:57.953449 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:57.953422 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7"] Sep 08 04:14:57.953683 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:57.953659 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7" podUID="15332324-579d-429d-97c0-6fbe37fea804" containerName="payload-processing" containerID="cri-o://926f047b28d3ac78ad23765e8a79b70bea60a9b09a06e6d19a46453bd3a2975c" gracePeriod=30 Sep 08 04:14:57.963575 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:57.962561 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn"] Sep 08 04:14:57.963575 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:57.962800 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn" podUID="c7057246-1d51-4e96-8f0f-487a16dfadd2" containerName="payload-pre-processing" containerID="cri-o://ee9fd64722f9125a586d2656890c8c110589127acf1aefbf97e17dd5a7240d43" gracePeriod=30 Sep 08 04:14:58.022174 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.022141 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:58.129620 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.129579 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:58.141664 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.141637 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cc32bbe-gzhh9"] Sep 08 04:14:58.144256 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.144231 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4cc32bbe-gzhh9"] Sep 08 04:14:58.198292 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.198204 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:58.247361 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.247340 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7" Sep 08 04:14:58.252833 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.252727 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn" Sep 08 04:14:58.294643 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.294615 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:58.408580 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.408545 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15332324-579d-429d-97c0-6fbe37fea804-plugins-config-volume\") pod \"15332324-579d-429d-97c0-6fbe37fea804\" (UID: \"15332324-579d-429d-97c0-6fbe37fea804\") " Sep 08 04:14:58.408742 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.408591 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qg4d6\" (UniqueName: \"kubernetes.io/projected/15332324-579d-429d-97c0-6fbe37fea804-kube-api-access-qg4d6\") pod \"15332324-579d-429d-97c0-6fbe37fea804\" (UID: \"15332324-579d-429d-97c0-6fbe37fea804\") " Sep 08 04:14:58.408742 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.408612 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqr4q\" (UniqueName: \"kubernetes.io/projected/c7057246-1d51-4e96-8f0f-487a16dfadd2-kube-api-access-lqr4q\") pod \"c7057246-1d51-4e96-8f0f-487a16dfadd2\" (UID: \"c7057246-1d51-4e96-8f0f-487a16dfadd2\") " Sep 08 04:14:58.408742 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.408650 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7057246-1d51-4e96-8f0f-487a16dfadd2-plugins-config-volume\") pod \"c7057246-1d51-4e96-8f0f-487a16dfadd2\" (UID: \"c7057246-1d51-4e96-8f0f-487a16dfadd2\") " Sep 08 04:14:58.408947 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.408920 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15332324-579d-429d-97c0-6fbe37fea804-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "15332324-579d-429d-97c0-6fbe37fea804" (UID: "15332324-579d-429d-97c0-6fbe37fea804"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:14:58.409057 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.409037 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7057246-1d51-4e96-8f0f-487a16dfadd2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c7057246-1d51-4e96-8f0f-487a16dfadd2" (UID: "c7057246-1d51-4e96-8f0f-487a16dfadd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:14:58.410903 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.410880 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15332324-579d-429d-97c0-6fbe37fea804-kube-api-access-qg4d6" (OuterVolumeSpecName: "kube-api-access-qg4d6") pod "15332324-579d-429d-97c0-6fbe37fea804" (UID: "15332324-579d-429d-97c0-6fbe37fea804"). InnerVolumeSpecName "kube-api-access-qg4d6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:14:58.410976 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.410915 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7057246-1d51-4e96-8f0f-487a16dfadd2-kube-api-access-lqr4q" (OuterVolumeSpecName: "kube-api-access-lqr4q") pod "c7057246-1d51-4e96-8f0f-487a16dfadd2" (UID: "c7057246-1d51-4e96-8f0f-487a16dfadd2"). InnerVolumeSpecName "kube-api-access-lqr4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:14:58.509536 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.509482 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15332324-579d-429d-97c0-6fbe37fea804-plugins-config-volume\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:14:58.509536 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.509534 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg4d6\" (UniqueName: \"kubernetes.io/projected/15332324-579d-429d-97c0-6fbe37fea804-kube-api-access-qg4d6\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:14:58.509681 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.509549 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqr4q\" (UniqueName: \"kubernetes.io/projected/c7057246-1d51-4e96-8f0f-487a16dfadd2-kube-api-access-lqr4q\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:14:58.509681 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.509562 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7057246-1d51-4e96-8f0f-487a16dfadd2-plugins-config-volume\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:14:58.567746 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.567723 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0aa30b0-d047-4617-a435-5229ab88e850" path="/var/lib/kubelet/pods/e0aa30b0-d047-4617-a435-5229ab88e850/volumes" Sep 08 04:14:58.602409 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.602386 2574 generic.go:356] "Generic (PLEG): container finished" podID="15332324-579d-429d-97c0-6fbe37fea804" containerID="926f047b28d3ac78ad23765e8a79b70bea60a9b09a06e6d19a46453bd3a2975c" exitCode=0 Sep 08 04:14:58.602495 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.602438 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7" Sep 08 04:14:58.602495 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.602469 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7" event={"ID":"15332324-579d-429d-97c0-6fbe37fea804","Type":"ContainerDied","Data":"926f047b28d3ac78ad23765e8a79b70bea60a9b09a06e6d19a46453bd3a2975c"} Sep 08 04:14:58.602628 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.602543 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7" event={"ID":"15332324-579d-429d-97c0-6fbe37fea804","Type":"ContainerDied","Data":"007a1562d96ca5f675ef8295f0746c1fe1bda8cf8e701c2d0381c2a83ffb3842"} Sep 08 04:14:58.602628 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.602573 2574 scope.go:117] "RemoveContainer" containerID="926f047b28d3ac78ad23765e8a79b70bea60a9b09a06e6d19a46453bd3a2975c" Sep 08 04:14:58.603902 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.603871 2574 generic.go:356] "Generic (PLEG): container finished" podID="c7057246-1d51-4e96-8f0f-487a16dfadd2" containerID="ee9fd64722f9125a586d2656890c8c110589127acf1aefbf97e17dd5a7240d43" exitCode=0 Sep 08 04:14:58.604004 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.603910 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn" event={"ID":"c7057246-1d51-4e96-8f0f-487a16dfadd2","Type":"ContainerDied","Data":"ee9fd64722f9125a586d2656890c8c110589127acf1aefbf97e17dd5a7240d43"} Sep 08 04:14:58.604004 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.603927 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn" Sep 08 04:14:58.604004 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.603960 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn" event={"ID":"c7057246-1d51-4e96-8f0f-487a16dfadd2","Type":"ContainerDied","Data":"969a0dcb9a7c8e8d14b5b460c03cda45301002241761e3c98fb2ad214f884372"} Sep 08 04:14:58.611394 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.611377 2574 scope.go:117] "RemoveContainer" containerID="926f047b28d3ac78ad23765e8a79b70bea60a9b09a06e6d19a46453bd3a2975c" Sep 08 04:14:58.611678 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:14:58.611657 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"926f047b28d3ac78ad23765e8a79b70bea60a9b09a06e6d19a46453bd3a2975c\": container with ID starting with 926f047b28d3ac78ad23765e8a79b70bea60a9b09a06e6d19a46453bd3a2975c not found: ID does not exist" containerID="926f047b28d3ac78ad23765e8a79b70bea60a9b09a06e6d19a46453bd3a2975c" Sep 08 04:14:58.611772 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.611683 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"926f047b28d3ac78ad23765e8a79b70bea60a9b09a06e6d19a46453bd3a2975c"} err="failed to get container status \"926f047b28d3ac78ad23765e8a79b70bea60a9b09a06e6d19a46453bd3a2975c\": rpc error: code = NotFound desc = could not find container \"926f047b28d3ac78ad23765e8a79b70bea60a9b09a06e6d19a46453bd3a2975c\": container with ID starting with 926f047b28d3ac78ad23765e8a79b70bea60a9b09a06e6d19a46453bd3a2975c not found: ID does not exist" Sep 08 04:14:58.611772 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.611700 2574 scope.go:117] "RemoveContainer" containerID="ee9fd64722f9125a586d2656890c8c110589127acf1aefbf97e17dd5a7240d43" Sep 08 04:14:58.619744 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.619728 2574 scope.go:117] "RemoveContainer" containerID="ee9fd64722f9125a586d2656890c8c110589127acf1aefbf97e17dd5a7240d43" Sep 08 04:14:58.620038 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:14:58.620014 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee9fd64722f9125a586d2656890c8c110589127acf1aefbf97e17dd5a7240d43\": container with ID starting with ee9fd64722f9125a586d2656890c8c110589127acf1aefbf97e17dd5a7240d43 not found: ID does not exist" containerID="ee9fd64722f9125a586d2656890c8c110589127acf1aefbf97e17dd5a7240d43" Sep 08 04:14:58.620154 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.620043 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee9fd64722f9125a586d2656890c8c110589127acf1aefbf97e17dd5a7240d43"} err="failed to get container status \"ee9fd64722f9125a586d2656890c8c110589127acf1aefbf97e17dd5a7240d43\": rpc error: code = NotFound desc = could not find container \"ee9fd64722f9125a586d2656890c8c110589127acf1aefbf97e17dd5a7240d43\": container with ID starting with ee9fd64722f9125a586d2656890c8c110589127acf1aefbf97e17dd5a7240d43 not found: ID does not exist" Sep 08 04:14:58.621202 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.621179 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7"] Sep 08 04:14:58.624753 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.624733 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4cc32bbe-848d7bdc5d-jw4d7"] Sep 08 04:14:58.634113 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.634094 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn"] Sep 08 04:14:58.637941 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.637920 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4cc32bbe-987985c4b-45cgn"] Sep 08 04:14:58.711260 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:58.711232 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:59.116907 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:59.116881 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:14:59.212614 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:14:59.212585 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:15:00.139153 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.139107 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-f0271e-29814015-v8wvj"] Sep 08 04:15:00.139544 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.139522 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7057246-1d51-4e96-8f0f-487a16dfadd2" containerName="payload-pre-processing" Sep 08 04:15:00.139544 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.139538 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7057246-1d51-4e96-8f0f-487a16dfadd2" containerName="payload-pre-processing" Sep 08 04:15:00.139634 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.139558 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0aa30b0-d047-4617-a435-5229ab88e850" containerName="revoke-keys" Sep 08 04:15:00.139634 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.139564 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0aa30b0-d047-4617-a435-5229ab88e850" containerName="revoke-keys" Sep 08 04:15:00.139634 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.139574 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15332324-579d-429d-97c0-6fbe37fea804" containerName="payload-processing" Sep 08 04:15:00.139634 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.139580 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="15332324-579d-429d-97c0-6fbe37fea804" containerName="payload-processing" Sep 08 04:15:00.139768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.139646 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="15332324-579d-429d-97c0-6fbe37fea804" containerName="payload-processing" Sep 08 04:15:00.139768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.139655 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7057246-1d51-4e96-8f0f-487a16dfadd2" containerName="payload-pre-processing" Sep 08 04:15:00.139768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.139663 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0aa30b0-d047-4617-a435-5229ab88e850" containerName="revoke-keys" Sep 08 04:15:00.143980 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.143960 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814015-2jplb"] Sep 08 04:15:00.144091 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.144073 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-f0271e-29814015-v8wvj" Sep 08 04:15:00.146392 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.146370 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7r29w\"" Sep 08 04:15:00.146889 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.146875 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ff0f3e-29814015-qqvnl"] Sep 08 04:15:00.147023 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.147003 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814015-2jplb" Sep 08 04:15:00.150612 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.150582 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-f0271e-29814015-v8wvj"] Sep 08 04:15:00.150612 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.150601 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ff0f3e-29814015-qqvnl" Sep 08 04:15:00.154624 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.154466 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814015-2jplb"] Sep 08 04:15:00.157915 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.157738 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ff0f3e-29814015-qqvnl"] Sep 08 04:15:00.159025 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.159005 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-f0271e-29814015-v8wvj" Sep 08 04:15:00.168487 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.168470 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814015-2jplb" Sep 08 04:15:00.206673 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.206648 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ff0f3e-29814015-qqvnl" Sep 08 04:15:00.444618 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.444588 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-f0271e-29814015-v8wvj"] Sep 08 04:15:00.446116 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:15:00.446086 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d3e936_1e88_4a73_a8c1_9fe9b2561531.slice/crio-0cb68c5467891e85727ee14989cad6b6231f4a772b54901bb1a746ff01383357 WatchSource:0}: Error finding container 0cb68c5467891e85727ee14989cad6b6231f4a772b54901bb1a746ff01383357: Status 404 returned error can't find the container with id 0cb68c5467891e85727ee14989cad6b6231f4a772b54901bb1a746ff01383357 Sep 08 04:15:00.460198 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.460173 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814015-2jplb"] Sep 08 04:15:00.461285 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:15:00.461263 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4247cb85_2393_4e48_8f8a_5d23d56d9a03.slice/crio-d6cd72f3f7538bcef7f69ad9e1b24422da06312f649bf235feb6d475a55f4d2f WatchSource:0}: Error finding container d6cd72f3f7538bcef7f69ad9e1b24422da06312f649bf235feb6d475a55f4d2f: Status 404 returned error can't find the container with id d6cd72f3f7538bcef7f69ad9e1b24422da06312f649bf235feb6d475a55f4d2f Sep 08 04:15:00.509560 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.509536 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ff0f3e-29814015-qqvnl"] Sep 08 04:15:00.510201 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:15:00.510176 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40f5758_975c_47cc_9e91_095556297aa6.slice/crio-db43f815b847b2496a4cb5b6284e60fd94503aae725a82f4318445398f7fc20a WatchSource:0}: Error finding container db43f815b847b2496a4cb5b6284e60fd94503aae725a82f4318445398f7fc20a: Status 404 returned error can't find the container with id db43f815b847b2496a4cb5b6284e60fd94503aae725a82f4318445398f7fc20a Sep 08 04:15:00.568268 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.568240 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15332324-579d-429d-97c0-6fbe37fea804" path="/var/lib/kubelet/pods/15332324-579d-429d-97c0-6fbe37fea804/volumes" Sep 08 04:15:00.568603 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.568590 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7057246-1d51-4e96-8f0f-487a16dfadd2" path="/var/lib/kubelet/pods/c7057246-1d51-4e96-8f0f-487a16dfadd2/volumes" Sep 08 04:15:00.619015 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.618991 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814015-2jplb" event={"ID":"4247cb85-2393-4e48-8f8a-5d23d56d9a03","Type":"ContainerStarted","Data":"d6cd72f3f7538bcef7f69ad9e1b24422da06312f649bf235feb6d475a55f4d2f"} Sep 08 04:15:00.620758 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.620642 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-f0271e-29814015-v8wvj" event={"ID":"36d3e936-1e88-4a73-a8c1-9fe9b2561531","Type":"ContainerStarted","Data":"0cb68c5467891e85727ee14989cad6b6231f4a772b54901bb1a746ff01383357"} Sep 08 04:15:00.621903 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:00.621884 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ff0f3e-29814015-qqvnl" event={"ID":"c40f5758-975c-47cc-9e91-095556297aa6","Type":"ContainerStarted","Data":"db43f815b847b2496a4cb5b6284e60fd94503aae725a82f4318445398f7fc20a"} Sep 08 04:15:01.580929 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:01.580880 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:15:01.634312 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:01.634267 2574 generic.go:356] "Generic (PLEG): container finished" podID="4247cb85-2393-4e48-8f8a-5d23d56d9a03" containerID="92302392b7c22620ee9549662cad3f7d36bc89af910b3d33d4311435e5406222" exitCode=0 Sep 08 04:15:01.634429 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:01.634359 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814015-2jplb" event={"ID":"4247cb85-2393-4e48-8f8a-5d23d56d9a03","Type":"ContainerDied","Data":"92302392b7c22620ee9549662cad3f7d36bc89af910b3d33d4311435e5406222"} Sep 08 04:15:01.635904 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:01.635876 2574 generic.go:356] "Generic (PLEG): container finished" podID="36d3e936-1e88-4a73-a8c1-9fe9b2561531" containerID="bf88e0a8cb10a4699793ee389e8897617deee39cb2767c77c9a90a90ebc4c55f" exitCode=0 Sep 08 04:15:01.636008 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:01.635959 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-f0271e-29814015-v8wvj" event={"ID":"36d3e936-1e88-4a73-a8c1-9fe9b2561531","Type":"ContainerDied","Data":"bf88e0a8cb10a4699793ee389e8897617deee39cb2767c77c9a90a90ebc4c55f"} Sep 08 04:15:01.637473 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:01.637453 2574 generic.go:356] "Generic (PLEG): container finished" podID="c40f5758-975c-47cc-9e91-095556297aa6" containerID="9f1c4940536c429df81d9391e00724fc26280035c5c811571d9e4bfcee98cb38" exitCode=0 Sep 08 04:15:01.637587 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:01.637537 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ff0f3e-29814015-qqvnl" event={"ID":"c40f5758-975c-47cc-9e91-095556297aa6","Type":"ContainerDied","Data":"9f1c4940536c429df81d9391e00724fc26280035c5c811571d9e4bfcee98cb38"} Sep 08 04:15:01.655383 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:01.655330 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:15:02.733963 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:02.733933 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:15:02.835177 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:02.835154 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:15:02.840077 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:02.840056 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-f0271e-29814015-v8wvj" Sep 08 04:15:02.843219 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:02.843202 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814015-2jplb" Sep 08 04:15:02.846248 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:02.846230 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ff0f3e-29814015-qqvnl" Sep 08 04:15:02.935075 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:02.935010 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:15:03.219206 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.219143 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:15:03.294849 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.294823 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:15:03.322250 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.322221 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw"] Sep 08 04:15:03.322694 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.322672 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c40f5758-975c-47cc-9e91-095556297aa6" containerName="cleanup" Sep 08 04:15:03.322694 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.322691 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c40f5758-975c-47cc-9e91-095556297aa6" containerName="cleanup" Sep 08 04:15:03.322848 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.322702 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36d3e936-1e88-4a73-a8c1-9fe9b2561531" containerName="cleanup" Sep 08 04:15:03.322848 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.322708 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="36d3e936-1e88-4a73-a8c1-9fe9b2561531" containerName="cleanup" Sep 08 04:15:03.322848 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.322749 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4247cb85-2393-4e48-8f8a-5d23d56d9a03" containerName="cleanup" Sep 08 04:15:03.322848 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.322757 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4247cb85-2393-4e48-8f8a-5d23d56d9a03" containerName="cleanup" Sep 08 04:15:03.322848 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.322842 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="36d3e936-1e88-4a73-a8c1-9fe9b2561531" containerName="cleanup" Sep 08 04:15:03.323029 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.322855 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4247cb85-2393-4e48-8f8a-5d23d56d9a03" containerName="cleanup" Sep 08 04:15:03.323029 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.322867 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c40f5758-975c-47cc-9e91-095556297aa6" containerName="cleanup" Sep 08 04:15:03.325840 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.325826 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw" Sep 08 04:15:03.328379 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.328358 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4001bd59\"" Sep 08 04:15:03.328485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.328444 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4001bd59-dockercfg-9s587\"" Sep 08 04:15:03.336232 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.336211 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw"] Sep 08 04:15:03.362111 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.362087 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x"] Sep 08 04:15:03.365364 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.365347 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x" Sep 08 04:15:03.373458 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.373428 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x"] Sep 08 04:15:03.457695 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.457632 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wghb\" (UniqueName: \"kubernetes.io/projected/f1b17f6c-ec63-4d8c-8ee9-550404938387-kube-api-access-9wghb\") pod \"payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw\" (UID: \"f1b17f6c-ec63-4d8c-8ee9-550404938387\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw" Sep 08 04:15:03.457887 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.457780 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1b17f6c-ec63-4d8c-8ee9-550404938387-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw\" (UID: \"f1b17f6c-ec63-4d8c-8ee9-550404938387\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw" Sep 08 04:15:03.457887 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.457834 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz7gf\" (UniqueName: \"kubernetes.io/projected/79d53266-e760-49d4-8af3-47dade60ae81-kube-api-access-nz7gf\") pod \"payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x\" (UID: \"79d53266-e760-49d4-8af3-47dade60ae81\") " pod="openshift-ingress/payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x" Sep 08 04:15:03.457887 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.457862 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d53266-e760-49d4-8af3-47dade60ae81-plugins-config-volume\") pod \"payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x\" (UID: \"79d53266-e760-49d4-8af3-47dade60ae81\") " pod="openshift-ingress/payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x" Sep 08 04:15:03.558523 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.558477 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1b17f6c-ec63-4d8c-8ee9-550404938387-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw\" (UID: \"f1b17f6c-ec63-4d8c-8ee9-550404938387\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw" Sep 08 04:15:03.558698 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.558539 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz7gf\" (UniqueName: \"kubernetes.io/projected/79d53266-e760-49d4-8af3-47dade60ae81-kube-api-access-nz7gf\") pod \"payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x\" (UID: \"79d53266-e760-49d4-8af3-47dade60ae81\") " pod="openshift-ingress/payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x" Sep 08 04:15:03.558698 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.558565 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d53266-e760-49d4-8af3-47dade60ae81-plugins-config-volume\") pod \"payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x\" (UID: \"79d53266-e760-49d4-8af3-47dade60ae81\") " pod="openshift-ingress/payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x" Sep 08 04:15:03.558698 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.558594 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wghb\" (UniqueName: \"kubernetes.io/projected/f1b17f6c-ec63-4d8c-8ee9-550404938387-kube-api-access-9wghb\") pod \"payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw\" (UID: \"f1b17f6c-ec63-4d8c-8ee9-550404938387\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw" Sep 08 04:15:03.559392 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.559351 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d53266-e760-49d4-8af3-47dade60ae81-plugins-config-volume\") pod \"payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x\" (UID: \"79d53266-e760-49d4-8af3-47dade60ae81\") " pod="openshift-ingress/payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x" Sep 08 04:15:03.559539 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.559400 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1b17f6c-ec63-4d8c-8ee9-550404938387-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw\" (UID: \"f1b17f6c-ec63-4d8c-8ee9-550404938387\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw" Sep 08 04:15:03.566291 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.566267 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wghb\" (UniqueName: \"kubernetes.io/projected/f1b17f6c-ec63-4d8c-8ee9-550404938387-kube-api-access-9wghb\") pod \"payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw\" (UID: \"f1b17f6c-ec63-4d8c-8ee9-550404938387\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw" Sep 08 04:15:03.566423 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.566351 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz7gf\" (UniqueName: \"kubernetes.io/projected/79d53266-e760-49d4-8af3-47dade60ae81-kube-api-access-nz7gf\") pod \"payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x\" (UID: \"79d53266-e760-49d4-8af3-47dade60ae81\") " pod="openshift-ingress/payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x" Sep 08 04:15:03.636187 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.636162 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw" Sep 08 04:15:03.651022 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.650994 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-f0271e-29814015-v8wvj" Sep 08 04:15:03.651022 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.651003 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-f0271e-29814015-v8wvj" event={"ID":"36d3e936-1e88-4a73-a8c1-9fe9b2561531","Type":"ContainerDied","Data":"0cb68c5467891e85727ee14989cad6b6231f4a772b54901bb1a746ff01383357"} Sep 08 04:15:03.651170 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.651049 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cb68c5467891e85727ee14989cad6b6231f4a772b54901bb1a746ff01383357" Sep 08 04:15:03.652806 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.652608 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ff0f3e-29814015-qqvnl" Sep 08 04:15:03.652806 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.652712 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ff0f3e-29814015-qqvnl" event={"ID":"c40f5758-975c-47cc-9e91-095556297aa6","Type":"ContainerDied","Data":"db43f815b847b2496a4cb5b6284e60fd94503aae725a82f4318445398f7fc20a"} Sep 08 04:15:03.652806 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.652770 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db43f815b847b2496a4cb5b6284e60fd94503aae725a82f4318445398f7fc20a" Sep 08 04:15:03.654712 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.654690 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814015-2jplb" event={"ID":"4247cb85-2393-4e48-8f8a-5d23d56d9a03","Type":"ContainerDied","Data":"d6cd72f3f7538bcef7f69ad9e1b24422da06312f649bf235feb6d475a55f4d2f"} Sep 08 04:15:03.654712 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.654709 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814015-2jplb" Sep 08 04:15:03.654849 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.654737 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6cd72f3f7538bcef7f69ad9e1b24422da06312f649bf235feb6d475a55f4d2f" Sep 08 04:15:03.675898 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.675879 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x" Sep 08 04:15:03.919112 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.919090 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw"] Sep 08 04:15:03.921275 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:15:03.921231 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b17f6c_ec63_4d8c_8ee9_550404938387.slice/crio-97f02fde8d808f40193647336d4df352b2ac8f700e9f98fdb45617a26eef326d WatchSource:0}: Error finding container 97f02fde8d808f40193647336d4df352b2ac8f700e9f98fdb45617a26eef326d: Status 404 returned error can't find the container with id 97f02fde8d808f40193647336d4df352b2ac8f700e9f98fdb45617a26eef326d Sep 08 04:15:03.967555 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:03.967528 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x"] Sep 08 04:15:03.967908 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:15:03.967881 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d53266_e760_49d4_8af3_47dade60ae81.slice/crio-f98c78bfd441c2a06120af9c98a5f51dbb257deaa0081470ab6637eb5885a34d WatchSource:0}: Error finding container f98c78bfd441c2a06120af9c98a5f51dbb257deaa0081470ab6637eb5885a34d: Status 404 returned error can't find the container with id f98c78bfd441c2a06120af9c98a5f51dbb257deaa0081470ab6637eb5885a34d Sep 08 04:15:04.661469 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:04.661425 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x" event={"ID":"79d53266-e760-49d4-8af3-47dade60ae81","Type":"ContainerStarted","Data":"b0497ae8990c4d11472b65b679442c75be1e03c74f20927dcb8c91788c106c7c"} Sep 08 04:15:04.661659 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:04.661495 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x" Sep 08 04:15:04.661659 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:04.661545 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x" event={"ID":"79d53266-e760-49d4-8af3-47dade60ae81","Type":"ContainerStarted","Data":"f98c78bfd441c2a06120af9c98a5f51dbb257deaa0081470ab6637eb5885a34d"} Sep 08 04:15:04.662900 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:04.662877 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw" event={"ID":"f1b17f6c-ec63-4d8c-8ee9-550404938387","Type":"ContainerStarted","Data":"9b529209d4f1eec0d90ff134f5315468005fbf526b0f074bdb16b5ca78f8bbff"} Sep 08 04:15:04.662982 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:04.662914 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw" event={"ID":"f1b17f6c-ec63-4d8c-8ee9-550404938387","Type":"ContainerStarted","Data":"97f02fde8d808f40193647336d4df352b2ac8f700e9f98fdb45617a26eef326d"} Sep 08 04:15:04.662982 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:04.662933 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw" Sep 08 04:15:04.678243 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:04.678199 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x" podStartSLOduration=1.6781871179999999 podStartE2EDuration="1.678187118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:15:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:15:04.67600467 +0000 UTC m=+878.598465429" watchObservedRunningTime="2026-09-08 04:15:04.678187118 +0000 UTC m=+878.600647879" Sep 08 04:15:04.691617 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:04.691573 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw" podStartSLOduration=1.691561162 podStartE2EDuration="1.691561162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:15:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:15:04.690132958 +0000 UTC m=+878.612593723" watchObservedRunningTime="2026-09-08 04:15:04.691561162 +0000 UTC m=+878.614021900" Sep 08 04:15:15.670312 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:15.670278 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw" Sep 08 04:15:15.670690 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:15.670347 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x" Sep 08 04:15:25.064684 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:25.064601 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:15:27.303485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:27.303445 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4001bd59-w6fp6"] Sep 08 04:15:27.309024 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:27.308999 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4001bd59-w6fp6" Sep 08 04:15:27.311938 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:27.311886 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7r29w\"" Sep 08 04:15:27.312830 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:27.312159 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 04:15:27.312830 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:27.312755 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4001bd59-w6fp6"] Sep 08 04:15:27.350626 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:27.350602 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db6e0f45-9da1-4ca7-a289-fc435b191d2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4001bd59-w6fp6\" (UID: \"db6e0f45-9da1-4ca7-a289-fc435b191d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4001bd59-w6fp6" Sep 08 04:15:27.451139 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:27.451115 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db6e0f45-9da1-4ca7-a289-fc435b191d2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4001bd59-w6fp6\" (UID: \"db6e0f45-9da1-4ca7-a289-fc435b191d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4001bd59-w6fp6" Sep 08 04:15:27.451704 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:27.451684 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db6e0f45-9da1-4ca7-a289-fc435b191d2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4001bd59-w6fp6\" (UID: \"db6e0f45-9da1-4ca7-a289-fc435b191d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4001bd59-w6fp6" Sep 08 04:15:27.622250 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:27.622197 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4001bd59-w6fp6" Sep 08 04:15:27.892062 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:27.892037 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4001bd59-w6fp6"] Sep 08 04:15:27.893201 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:15:27.893170 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb6e0f45_9da1_4ca7_a289_fc435b191d2c.slice/crio-2ee40e41973771a242c726dcfa83acb4c7ce42b9addf608b9d8089c93e7a066a WatchSource:0}: Error finding container 2ee40e41973771a242c726dcfa83acb4c7ce42b9addf608b9d8089c93e7a066a: Status 404 returned error can't find the container with id 2ee40e41973771a242c726dcfa83acb4c7ce42b9addf608b9d8089c93e7a066a Sep 08 04:15:28.810327 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:28.810265 2574 generic.go:356] "Generic (PLEG): container finished" podID="db6e0f45-9da1-4ca7-a289-fc435b191d2c" containerID="4a464b67a054d9b9de29ac6df3c0a96d0363078189d696a2237317990ef14e7b" exitCode=0 Sep 08 04:15:28.810644 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:28.810347 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4001bd59-w6fp6" event={"ID":"db6e0f45-9da1-4ca7-a289-fc435b191d2c","Type":"ContainerDied","Data":"4a464b67a054d9b9de29ac6df3c0a96d0363078189d696a2237317990ef14e7b"} Sep 08 04:15:28.810644 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:28.810402 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4001bd59-w6fp6" event={"ID":"db6e0f45-9da1-4ca7-a289-fc435b191d2c","Type":"ContainerStarted","Data":"2ee40e41973771a242c726dcfa83acb4c7ce42b9addf608b9d8089c93e7a066a"} Sep 08 04:15:29.949107 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:29.949085 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4001bd59-w6fp6" Sep 08 04:15:29.971877 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:29.971855 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db6e0f45-9da1-4ca7-a289-fc435b191d2c-maas-api-service-ca\") pod \"db6e0f45-9da1-4ca7-a289-fc435b191d2c\" (UID: \"db6e0f45-9da1-4ca7-a289-fc435b191d2c\") " Sep 08 04:15:29.972170 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:29.972152 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db6e0f45-9da1-4ca7-a289-fc435b191d2c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "db6e0f45-9da1-4ca7-a289-fc435b191d2c" (UID: "db6e0f45-9da1-4ca7-a289-fc435b191d2c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:15:30.072607 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:30.072551 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db6e0f45-9da1-4ca7-a289-fc435b191d2c-maas-api-service-ca\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:15:30.822485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:30.822451 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4001bd59-w6fp6" Sep 08 04:15:30.822485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:30.822479 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4001bd59-w6fp6" event={"ID":"db6e0f45-9da1-4ca7-a289-fc435b191d2c","Type":"ContainerDied","Data":"2ee40e41973771a242c726dcfa83acb4c7ce42b9addf608b9d8089c93e7a066a"} Sep 08 04:15:30.822694 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:30.822543 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ee40e41973771a242c726dcfa83acb4c7ce42b9addf608b9d8089c93e7a066a" Sep 08 04:15:33.227479 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.227449 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:15:33.262453 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.262420 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x"] Sep 08 04:15:33.262748 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.262714 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x" podUID="79d53266-e760-49d4-8af3-47dade60ae81" containerName="payload-processing" containerID="cri-o://b0497ae8990c4d11472b65b679442c75be1e03c74f20927dcb8c91788c106c7c" gracePeriod=30 Sep 08 04:15:33.269082 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.269037 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw"] Sep 08 04:15:33.269279 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.269249 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw" podUID="f1b17f6c-ec63-4d8c-8ee9-550404938387" containerName="payload-pre-processing" containerID="cri-o://9b529209d4f1eec0d90ff134f5315468005fbf526b0f074bdb16b5ca78f8bbff" gracePeriod=30 Sep 08 04:15:33.338060 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.338032 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:15:33.450845 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.450641 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:15:33.543345 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.543316 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:15:33.553601 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.553580 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x" Sep 08 04:15:33.556695 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.556661 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw" Sep 08 04:15:33.558330 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.558303 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4001bd59-w6fp6"] Sep 08 04:15:33.561772 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.561751 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4001bd59-w6fp6"] Sep 08 04:15:33.602764 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.602742 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nz7gf\" (UniqueName: \"kubernetes.io/projected/79d53266-e760-49d4-8af3-47dade60ae81-kube-api-access-nz7gf\") pod \"79d53266-e760-49d4-8af3-47dade60ae81\" (UID: \"79d53266-e760-49d4-8af3-47dade60ae81\") " Sep 08 04:15:33.602874 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.602776 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d53266-e760-49d4-8af3-47dade60ae81-plugins-config-volume\") pod \"79d53266-e760-49d4-8af3-47dade60ae81\" (UID: \"79d53266-e760-49d4-8af3-47dade60ae81\") " Sep 08 04:15:33.602949 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.602868 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wghb\" (UniqueName: \"kubernetes.io/projected/f1b17f6c-ec63-4d8c-8ee9-550404938387-kube-api-access-9wghb\") pod \"f1b17f6c-ec63-4d8c-8ee9-550404938387\" (UID: \"f1b17f6c-ec63-4d8c-8ee9-550404938387\") " Sep 08 04:15:33.602949 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.602916 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1b17f6c-ec63-4d8c-8ee9-550404938387-plugins-config-volume\") pod \"f1b17f6c-ec63-4d8c-8ee9-550404938387\" (UID: \"f1b17f6c-ec63-4d8c-8ee9-550404938387\") " Sep 08 04:15:33.603253 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.603225 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d53266-e760-49d4-8af3-47dade60ae81-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "79d53266-e760-49d4-8af3-47dade60ae81" (UID: "79d53266-e760-49d4-8af3-47dade60ae81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:15:33.603339 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.603300 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1b17f6c-ec63-4d8c-8ee9-550404938387-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f1b17f6c-ec63-4d8c-8ee9-550404938387" (UID: "f1b17f6c-ec63-4d8c-8ee9-550404938387"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:15:33.603765 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.603744 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d53266-e760-49d4-8af3-47dade60ae81-plugins-config-volume\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:15:33.603871 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.603770 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1b17f6c-ec63-4d8c-8ee9-550404938387-plugins-config-volume\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:15:33.605067 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.605041 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d53266-e760-49d4-8af3-47dade60ae81-kube-api-access-nz7gf" (OuterVolumeSpecName: "kube-api-access-nz7gf") pod "79d53266-e760-49d4-8af3-47dade60ae81" (UID: "79d53266-e760-49d4-8af3-47dade60ae81"). InnerVolumeSpecName "kube-api-access-nz7gf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:15:33.605149 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.605085 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1b17f6c-ec63-4d8c-8ee9-550404938387-kube-api-access-9wghb" (OuterVolumeSpecName: "kube-api-access-9wghb") pod "f1b17f6c-ec63-4d8c-8ee9-550404938387" (UID: "f1b17f6c-ec63-4d8c-8ee9-550404938387"). InnerVolumeSpecName "kube-api-access-9wghb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:15:33.610674 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.610643 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:15:33.704539 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.704499 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz7gf\" (UniqueName: \"kubernetes.io/projected/79d53266-e760-49d4-8af3-47dade60ae81-kube-api-access-nz7gf\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:15:33.704539 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.704539 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wghb\" (UniqueName: \"kubernetes.io/projected/f1b17f6c-ec63-4d8c-8ee9-550404938387-kube-api-access-9wghb\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:15:33.844931 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.844855 2574 generic.go:356] "Generic (PLEG): container finished" podID="f1b17f6c-ec63-4d8c-8ee9-550404938387" containerID="9b529209d4f1eec0d90ff134f5315468005fbf526b0f074bdb16b5ca78f8bbff" exitCode=0 Sep 08 04:15:33.844931 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.844898 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw" Sep 08 04:15:33.845116 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.844934 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw" event={"ID":"f1b17f6c-ec63-4d8c-8ee9-550404938387","Type":"ContainerDied","Data":"9b529209d4f1eec0d90ff134f5315468005fbf526b0f074bdb16b5ca78f8bbff"} Sep 08 04:15:33.845116 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.844991 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw" event={"ID":"f1b17f6c-ec63-4d8c-8ee9-550404938387","Type":"ContainerDied","Data":"97f02fde8d808f40193647336d4df352b2ac8f700e9f98fdb45617a26eef326d"} Sep 08 04:15:33.845116 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.845015 2574 scope.go:117] "RemoveContainer" containerID="9b529209d4f1eec0d90ff134f5315468005fbf526b0f074bdb16b5ca78f8bbff" Sep 08 04:15:33.846455 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.846432 2574 generic.go:356] "Generic (PLEG): container finished" podID="79d53266-e760-49d4-8af3-47dade60ae81" containerID="b0497ae8990c4d11472b65b679442c75be1e03c74f20927dcb8c91788c106c7c" exitCode=0 Sep 08 04:15:33.846586 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.846471 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x" Sep 08 04:15:33.846586 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.846527 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x" event={"ID":"79d53266-e760-49d4-8af3-47dade60ae81","Type":"ContainerDied","Data":"b0497ae8990c4d11472b65b679442c75be1e03c74f20927dcb8c91788c106c7c"} Sep 08 04:15:33.846586 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.846582 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x" event={"ID":"79d53266-e760-49d4-8af3-47dade60ae81","Type":"ContainerDied","Data":"f98c78bfd441c2a06120af9c98a5f51dbb257deaa0081470ab6637eb5885a34d"} Sep 08 04:15:33.855348 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.855249 2574 scope.go:117] "RemoveContainer" containerID="9b529209d4f1eec0d90ff134f5315468005fbf526b0f074bdb16b5ca78f8bbff" Sep 08 04:15:33.855579 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:15:33.855558 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b529209d4f1eec0d90ff134f5315468005fbf526b0f074bdb16b5ca78f8bbff\": container with ID starting with 9b529209d4f1eec0d90ff134f5315468005fbf526b0f074bdb16b5ca78f8bbff not found: ID does not exist" containerID="9b529209d4f1eec0d90ff134f5315468005fbf526b0f074bdb16b5ca78f8bbff" Sep 08 04:15:33.855637 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.855586 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b529209d4f1eec0d90ff134f5315468005fbf526b0f074bdb16b5ca78f8bbff"} err="failed to get container status \"9b529209d4f1eec0d90ff134f5315468005fbf526b0f074bdb16b5ca78f8bbff\": rpc error: code = NotFound desc = could not find container \"9b529209d4f1eec0d90ff134f5315468005fbf526b0f074bdb16b5ca78f8bbff\": container with ID starting with 9b529209d4f1eec0d90ff134f5315468005fbf526b0f074bdb16b5ca78f8bbff not found: ID does not exist" Sep 08 04:15:33.855637 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.855606 2574 scope.go:117] "RemoveContainer" containerID="b0497ae8990c4d11472b65b679442c75be1e03c74f20927dcb8c91788c106c7c" Sep 08 04:15:33.863607 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.863587 2574 scope.go:117] "RemoveContainer" containerID="b0497ae8990c4d11472b65b679442c75be1e03c74f20927dcb8c91788c106c7c" Sep 08 04:15:33.863861 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:15:33.863842 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0497ae8990c4d11472b65b679442c75be1e03c74f20927dcb8c91788c106c7c\": container with ID starting with b0497ae8990c4d11472b65b679442c75be1e03c74f20927dcb8c91788c106c7c not found: ID does not exist" containerID="b0497ae8990c4d11472b65b679442c75be1e03c74f20927dcb8c91788c106c7c" Sep 08 04:15:33.863923 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.863866 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0497ae8990c4d11472b65b679442c75be1e03c74f20927dcb8c91788c106c7c"} err="failed to get container status \"b0497ae8990c4d11472b65b679442c75be1e03c74f20927dcb8c91788c106c7c\": rpc error: code = NotFound desc = could not find container \"b0497ae8990c4d11472b65b679442c75be1e03c74f20927dcb8c91788c106c7c\": container with ID starting with b0497ae8990c4d11472b65b679442c75be1e03c74f20927dcb8c91788c106c7c not found: ID does not exist" Sep 08 04:15:33.889112 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.889086 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw"] Sep 08 04:15:33.909843 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.909824 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4001bd59-9cb576846-lm5dw"] Sep 08 04:15:33.926232 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.926211 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x"] Sep 08 04:15:33.931599 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:33.931579 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4001bd59-6b95f9865c-j962x"] Sep 08 04:15:34.568293 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:34.568239 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79d53266-e760-49d4-8af3-47dade60ae81" path="/var/lib/kubelet/pods/79d53266-e760-49d4-8af3-47dade60ae81/volumes" Sep 08 04:15:34.568601 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:34.568564 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db6e0f45-9da1-4ca7-a289-fc435b191d2c" path="/var/lib/kubelet/pods/db6e0f45-9da1-4ca7-a289-fc435b191d2c/volumes" Sep 08 04:15:34.568878 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:34.568865 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1b17f6c-ec63-4d8c-8ee9-550404938387" path="/var/lib/kubelet/pods/f1b17f6c-ec63-4d8c-8ee9-550404938387/volumes" Sep 08 04:15:34.662195 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:34.662165 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:15:35.092414 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:35.092385 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:15:35.395807 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:35.395728 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:15:35.585370 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:35.585336 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:15:37.833266 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:37.833230 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:15:37.929424 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:37.929394 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:15:39.088940 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.088907 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:15:39.187954 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.187918 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:15:39.288419 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.288385 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:15:39.561058 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.561025 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:15:39.646768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.646732 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:15:39.707211 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.707177 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x"] Sep 08 04:15:39.707559 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.707547 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1b17f6c-ec63-4d8c-8ee9-550404938387" containerName="payload-pre-processing" Sep 08 04:15:39.707600 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.707561 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1b17f6c-ec63-4d8c-8ee9-550404938387" containerName="payload-pre-processing" Sep 08 04:15:39.707600 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.707573 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db6e0f45-9da1-4ca7-a289-fc435b191d2c" containerName="revoke-keys" Sep 08 04:15:39.707600 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.707580 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="db6e0f45-9da1-4ca7-a289-fc435b191d2c" containerName="revoke-keys" Sep 08 04:15:39.707600 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.707590 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79d53266-e760-49d4-8af3-47dade60ae81" containerName="payload-processing" Sep 08 04:15:39.707600 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.707597 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="79d53266-e760-49d4-8af3-47dade60ae81" containerName="payload-processing" Sep 08 04:15:39.707762 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.707659 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="db6e0f45-9da1-4ca7-a289-fc435b191d2c" containerName="revoke-keys" Sep 08 04:15:39.707762 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.707668 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1b17f6c-ec63-4d8c-8ee9-550404938387" containerName="payload-pre-processing" Sep 08 04:15:39.707762 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.707674 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="79d53266-e760-49d4-8af3-47dade60ae81" containerName="payload-processing" Sep 08 04:15:39.712190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.712174 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x" Sep 08 04:15:39.714649 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.714627 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-efa641f8\"" Sep 08 04:15:39.714771 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.714680 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-efa641f8-dockercfg-cvrsg\"" Sep 08 04:15:39.724063 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.724041 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x"] Sep 08 04:15:39.744157 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.744132 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5"] Sep 08 04:15:39.747447 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.747431 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5" Sep 08 04:15:39.757975 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.757954 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5"] Sep 08 04:15:39.763079 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.763058 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f1e5c6-b9de-4b06-88d2-0acb30a27c40-plugins-config-volume\") pod \"payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x\" (UID: \"a8f1e5c6-b9de-4b06-88d2-0acb30a27c40\") " pod="openshift-ingress/payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x" Sep 08 04:15:39.763208 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.763190 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqdkd\" (UniqueName: \"kubernetes.io/projected/a8f1e5c6-b9de-4b06-88d2-0acb30a27c40-kube-api-access-gqdkd\") pod \"payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x\" (UID: \"a8f1e5c6-b9de-4b06-88d2-0acb30a27c40\") " pod="openshift-ingress/payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x" Sep 08 04:15:39.864536 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.864439 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqdkd\" (UniqueName: \"kubernetes.io/projected/a8f1e5c6-b9de-4b06-88d2-0acb30a27c40-kube-api-access-gqdkd\") pod \"payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x\" (UID: \"a8f1e5c6-b9de-4b06-88d2-0acb30a27c40\") " pod="openshift-ingress/payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x" Sep 08 04:15:39.864536 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.864491 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ce41604-57e5-461a-b5d7-64deae23923f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5\" (UID: \"7ce41604-57e5-461a-b5d7-64deae23923f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5" Sep 08 04:15:39.864768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.864538 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f1e5c6-b9de-4b06-88d2-0acb30a27c40-plugins-config-volume\") pod \"payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x\" (UID: \"a8f1e5c6-b9de-4b06-88d2-0acb30a27c40\") " pod="openshift-ingress/payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x" Sep 08 04:15:39.864768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.864662 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cmkb\" (UniqueName: \"kubernetes.io/projected/7ce41604-57e5-461a-b5d7-64deae23923f-kube-api-access-4cmkb\") pod \"payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5\" (UID: \"7ce41604-57e5-461a-b5d7-64deae23923f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5" Sep 08 04:15:39.865164 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.865145 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f1e5c6-b9de-4b06-88d2-0acb30a27c40-plugins-config-volume\") pod \"payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x\" (UID: \"a8f1e5c6-b9de-4b06-88d2-0acb30a27c40\") " pod="openshift-ingress/payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x" Sep 08 04:15:39.872166 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.872143 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqdkd\" (UniqueName: \"kubernetes.io/projected/a8f1e5c6-b9de-4b06-88d2-0acb30a27c40-kube-api-access-gqdkd\") pod \"payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x\" (UID: \"a8f1e5c6-b9de-4b06-88d2-0acb30a27c40\") " pod="openshift-ingress/payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x" Sep 08 04:15:39.965639 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.965613 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ce41604-57e5-461a-b5d7-64deae23923f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5\" (UID: \"7ce41604-57e5-461a-b5d7-64deae23923f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5" Sep 08 04:15:39.965796 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.965650 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cmkb\" (UniqueName: \"kubernetes.io/projected/7ce41604-57e5-461a-b5d7-64deae23923f-kube-api-access-4cmkb\") pod \"payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5\" (UID: \"7ce41604-57e5-461a-b5d7-64deae23923f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5" Sep 08 04:15:39.966258 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.966239 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ce41604-57e5-461a-b5d7-64deae23923f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5\" (UID: \"7ce41604-57e5-461a-b5d7-64deae23923f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5" Sep 08 04:15:39.973575 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:39.973545 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cmkb\" (UniqueName: \"kubernetes.io/projected/7ce41604-57e5-461a-b5d7-64deae23923f-kube-api-access-4cmkb\") pod \"payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5\" (UID: \"7ce41604-57e5-461a-b5d7-64deae23923f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5" Sep 08 04:15:40.023253 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:40.023221 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x" Sep 08 04:15:40.057628 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:40.057602 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5" Sep 08 04:15:40.300133 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:40.300106 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x"] Sep 08 04:15:40.302458 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:15:40.302434 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f1e5c6_b9de_4b06_88d2_0acb30a27c40.slice/crio-17f62156e5904c766d2cf91cb62673e7084735fe94c5b77bf13d210d7a38ecc2 WatchSource:0}: Error finding container 17f62156e5904c766d2cf91cb62673e7084735fe94c5b77bf13d210d7a38ecc2: Status 404 returned error can't find the container with id 17f62156e5904c766d2cf91cb62673e7084735fe94c5b77bf13d210d7a38ecc2 Sep 08 04:15:40.345367 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:40.342263 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5"] Sep 08 04:15:40.890179 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:40.890142 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5" event={"ID":"7ce41604-57e5-461a-b5d7-64deae23923f","Type":"ContainerStarted","Data":"cec086022c17f55787310ceb5e791c9a421c3b70ec5e32a513df0027f1eb21e5"} Sep 08 04:15:40.890350 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:40.890193 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5" event={"ID":"7ce41604-57e5-461a-b5d7-64deae23923f","Type":"ContainerStarted","Data":"c2fa59ddd52f55df707c5802a7cc0716aa48e744b33c829d7a8c9e176f7c4df7"} Sep 08 04:15:40.890350 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:40.890301 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5" Sep 08 04:15:40.891992 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:40.891967 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x" event={"ID":"a8f1e5c6-b9de-4b06-88d2-0acb30a27c40","Type":"ContainerStarted","Data":"19c6b9bb9506ee352922116566f1c3f8975d915ba92c0ae680e03cacc099d3e9"} Sep 08 04:15:40.892109 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:40.892013 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x" event={"ID":"a8f1e5c6-b9de-4b06-88d2-0acb30a27c40","Type":"ContainerStarted","Data":"17f62156e5904c766d2cf91cb62673e7084735fe94c5b77bf13d210d7a38ecc2"} Sep 08 04:15:40.892109 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:40.892057 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x" Sep 08 04:15:40.909172 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:40.909124 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5" podStartSLOduration=1.909111354 podStartE2EDuration="1.909111354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:15:40.906286882 +0000 UTC m=+914.828747639" watchObservedRunningTime="2026-09-08 04:15:40.909111354 +0000 UTC m=+914.831572156" Sep 08 04:15:40.924352 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:40.924303 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x" podStartSLOduration=1.9242906400000002 podStartE2EDuration="1.92429064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:15:40.922768434 +0000 UTC m=+914.845229203" watchObservedRunningTime="2026-09-08 04:15:40.92429064 +0000 UTC m=+914.846751379" Sep 08 04:15:51.901272 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:51.901236 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x" Sep 08 04:15:51.901692 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:15:51.901308 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5" Sep 08 04:16:00.911467 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:00.911429 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0271e-lqbsh"] Sep 08 04:16:00.914937 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:00.914920 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0271e-lqbsh" Sep 08 04:16:00.917628 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:00.917604 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 04:16:00.917929 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:00.917912 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7r29w\"" Sep 08 04:16:00.921951 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:00.921926 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0271e-lqbsh"] Sep 08 04:16:01.050079 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:01.050047 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924f5e79-95f6-4f4e-b2d3-ec38c905aa76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-f0271e-lqbsh\" (UID: \"924f5e79-95f6-4f4e-b2d3-ec38c905aa76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0271e-lqbsh" Sep 08 04:16:01.150824 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:01.150795 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924f5e79-95f6-4f4e-b2d3-ec38c905aa76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-f0271e-lqbsh\" (UID: \"924f5e79-95f6-4f4e-b2d3-ec38c905aa76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0271e-lqbsh" Sep 08 04:16:01.151380 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:01.151361 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924f5e79-95f6-4f4e-b2d3-ec38c905aa76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-f0271e-lqbsh\" (UID: \"924f5e79-95f6-4f4e-b2d3-ec38c905aa76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0271e-lqbsh" Sep 08 04:16:01.227095 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:01.227038 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0271e-lqbsh" Sep 08 04:16:01.496047 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:01.496012 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0271e-lqbsh"] Sep 08 04:16:01.668034 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:01.668006 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:02.015770 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:02.015736 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0271e-lqbsh" event={"ID":"924f5e79-95f6-4f4e-b2d3-ec38c905aa76","Type":"ContainerStarted","Data":"c79aa2f6484dade242cb4f03389b42bed85abca655d90b4e5c3aef3a3e1c49d5"} Sep 08 04:16:02.016135 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:02.015789 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0271e-lqbsh" event={"ID":"924f5e79-95f6-4f4e-b2d3-ec38c905aa76","Type":"ContainerStarted","Data":"b69c59ba3a870e8f47f9572a0310a000ee185c9c37cae2290f886e736c21539a"} Sep 08 04:16:02.034480 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:02.034435 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0271e-lqbsh" podStartSLOduration=2.034420693 podStartE2EDuration="2.034420693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:16:02.03184365 +0000 UTC m=+935.954304391" watchObservedRunningTime="2026-09-08 04:16:02.034420693 +0000 UTC m=+935.956881433" Sep 08 04:16:02.649783 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:02.649753 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-efa641f8-8znb7"] Sep 08 04:16:02.653238 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:02.653222 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-efa641f8-8znb7" Sep 08 04:16:02.659312 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:02.659289 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-efa641f8-8znb7"] Sep 08 04:16:02.764551 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:02.764515 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e0436e3-fc27-4639-b9d0-e988ff5bb8fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-efa641f8-8znb7\" (UID: \"3e0436e3-fc27-4639-b9d0-e988ff5bb8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-efa641f8-8znb7" Sep 08 04:16:02.865883 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:02.865845 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e0436e3-fc27-4639-b9d0-e988ff5bb8fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-efa641f8-8znb7\" (UID: \"3e0436e3-fc27-4639-b9d0-e988ff5bb8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-efa641f8-8znb7" Sep 08 04:16:02.866433 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:02.866412 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e0436e3-fc27-4639-b9d0-e988ff5bb8fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-efa641f8-8znb7\" (UID: \"3e0436e3-fc27-4639-b9d0-e988ff5bb8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-efa641f8-8znb7" Sep 08 04:16:02.965834 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:02.965767 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-efa641f8-8znb7" Sep 08 04:16:03.029401 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:03.029373 2574 generic.go:356] "Generic (PLEG): container finished" podID="924f5e79-95f6-4f4e-b2d3-ec38c905aa76" containerID="c79aa2f6484dade242cb4f03389b42bed85abca655d90b4e5c3aef3a3e1c49d5" exitCode=0 Sep 08 04:16:03.029829 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:03.029488 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0271e-lqbsh" event={"ID":"924f5e79-95f6-4f4e-b2d3-ec38c905aa76","Type":"ContainerDied","Data":"c79aa2f6484dade242cb4f03389b42bed85abca655d90b4e5c3aef3a3e1c49d5"} Sep 08 04:16:03.066671 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:03.066644 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:03.240883 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:03.240854 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-efa641f8-8znb7"] Sep 08 04:16:03.241146 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:16:03.241124 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e0436e3_fc27_4639_b9d0_e988ff5bb8fa.slice/crio-1253eed9392194134aed69d3bcbbfdfad9106892fc0e39f02ff6e4285d22075c WatchSource:0}: Error finding container 1253eed9392194134aed69d3bcbbfdfad9106892fc0e39f02ff6e4285d22075c: Status 404 returned error can't find the container with id 1253eed9392194134aed69d3bcbbfdfad9106892fc0e39f02ff6e4285d22075c Sep 08 04:16:04.036779 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:04.036747 2574 generic.go:356] "Generic (PLEG): container finished" podID="3e0436e3-fc27-4639-b9d0-e988ff5bb8fa" containerID="7a781917418de65508da2eb7a2e2cff2dc1713b0d1db5b2b88988e22e1c407c8" exitCode=0 Sep 08 04:16:04.037149 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:04.036821 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-efa641f8-8znb7" event={"ID":"3e0436e3-fc27-4639-b9d0-e988ff5bb8fa","Type":"ContainerDied","Data":"7a781917418de65508da2eb7a2e2cff2dc1713b0d1db5b2b88988e22e1c407c8"} Sep 08 04:16:04.037149 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:04.036871 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-efa641f8-8znb7" event={"ID":"3e0436e3-fc27-4639-b9d0-e988ff5bb8fa","Type":"ContainerStarted","Data":"1253eed9392194134aed69d3bcbbfdfad9106892fc0e39f02ff6e4285d22075c"} Sep 08 04:16:04.166985 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:04.166962 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0271e-lqbsh" Sep 08 04:16:04.277282 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:04.277247 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924f5e79-95f6-4f4e-b2d3-ec38c905aa76-maas-api-service-ca\") pod \"924f5e79-95f6-4f4e-b2d3-ec38c905aa76\" (UID: \"924f5e79-95f6-4f4e-b2d3-ec38c905aa76\") " Sep 08 04:16:04.277688 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:04.277662 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924f5e79-95f6-4f4e-b2d3-ec38c905aa76-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "924f5e79-95f6-4f4e-b2d3-ec38c905aa76" (UID: "924f5e79-95f6-4f4e-b2d3-ec38c905aa76"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:16:04.375880 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:04.375847 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:04.378104 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:04.378082 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924f5e79-95f6-4f4e-b2d3-ec38c905aa76-maas-api-service-ca\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:16:05.043656 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:05.043632 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0271e-lqbsh" Sep 08 04:16:05.044025 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:05.043672 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0271e-lqbsh" event={"ID":"924f5e79-95f6-4f4e-b2d3-ec38c905aa76","Type":"ContainerDied","Data":"b69c59ba3a870e8f47f9572a0310a000ee185c9c37cae2290f886e736c21539a"} Sep 08 04:16:05.044025 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:05.043716 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b69c59ba3a870e8f47f9572a0310a000ee185c9c37cae2290f886e736c21539a" Sep 08 04:16:05.175488 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:05.175464 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-efa641f8-8znb7" Sep 08 04:16:05.285961 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:05.285936 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e0436e3-fc27-4639-b9d0-e988ff5bb8fa-maas-api-service-ca\") pod \"3e0436e3-fc27-4639-b9d0-e988ff5bb8fa\" (UID: \"3e0436e3-fc27-4639-b9d0-e988ff5bb8fa\") " Sep 08 04:16:05.286279 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:05.286256 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e0436e3-fc27-4639-b9d0-e988ff5bb8fa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3e0436e3-fc27-4639-b9d0-e988ff5bb8fa" (UID: "3e0436e3-fc27-4639-b9d0-e988ff5bb8fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:16:05.386668 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:05.386609 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e0436e3-fc27-4639-b9d0-e988ff5bb8fa-maas-api-service-ca\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:16:06.054844 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:06.054815 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-efa641f8-8znb7" Sep 08 04:16:06.055225 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:06.054844 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-efa641f8-8znb7" event={"ID":"3e0436e3-fc27-4639-b9d0-e988ff5bb8fa","Type":"ContainerDied","Data":"1253eed9392194134aed69d3bcbbfdfad9106892fc0e39f02ff6e4285d22075c"} Sep 08 04:16:06.055225 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:06.054893 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1253eed9392194134aed69d3bcbbfdfad9106892fc0e39f02ff6e4285d22075c" Sep 08 04:16:09.277840 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:09.277795 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:09.876887 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:09.876852 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:09.902471 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:09.902430 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x"] Sep 08 04:16:09.902770 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:09.902732 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x" podUID="a8f1e5c6-b9de-4b06-88d2-0acb30a27c40" containerName="payload-processing" containerID="cri-o://19c6b9bb9506ee352922116566f1c3f8975d915ba92c0ae680e03cacc099d3e9" gracePeriod=30 Sep 08 04:16:09.913086 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:09.913059 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5"] Sep 08 04:16:09.913287 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:09.913266 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5" podUID="7ce41604-57e5-461a-b5d7-64deae23923f" containerName="payload-pre-processing" containerID="cri-o://cec086022c17f55787310ceb5e791c9a421c3b70ec5e32a513df0027f1eb21e5" gracePeriod=30 Sep 08 04:16:09.947859 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:09.947813 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:10.009970 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:10.009937 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:10.079635 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:10.079483 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-efa641f8-8znb7"] Sep 08 04:16:10.079832 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:10.079736 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:10.082111 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:10.082086 2574 generic.go:356] "Generic (PLEG): container finished" podID="7ce41604-57e5-461a-b5d7-64deae23923f" containerID="cec086022c17f55787310ceb5e791c9a421c3b70ec5e32a513df0027f1eb21e5" exitCode=0 Sep 08 04:16:10.082290 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:10.082132 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5" event={"ID":"7ce41604-57e5-461a-b5d7-64deae23923f","Type":"ContainerDied","Data":"cec086022c17f55787310ceb5e791c9a421c3b70ec5e32a513df0027f1eb21e5"} Sep 08 04:16:10.082736 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:10.082712 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-efa641f8-8znb7"] Sep 08 04:16:10.084212 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:10.084191 2574 generic.go:356] "Generic (PLEG): container finished" podID="a8f1e5c6-b9de-4b06-88d2-0acb30a27c40" containerID="19c6b9bb9506ee352922116566f1c3f8975d915ba92c0ae680e03cacc099d3e9" exitCode=0 Sep 08 04:16:10.084316 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:10.084257 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x" event={"ID":"a8f1e5c6-b9de-4b06-88d2-0acb30a27c40","Type":"ContainerDied","Data":"19c6b9bb9506ee352922116566f1c3f8975d915ba92c0ae680e03cacc099d3e9"} Sep 08 04:16:10.131105 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:10.131015 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:10.204896 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:10.204871 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x" Sep 08 04:16:10.207688 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:10.207664 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5" Sep 08 04:16:10.252486 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:10.252457 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:10.336314 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:10.336259 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f1e5c6-b9de-4b06-88d2-0acb30a27c40-plugins-config-volume\") pod \"a8f1e5c6-b9de-4b06-88d2-0acb30a27c40\" (UID: \"a8f1e5c6-b9de-4b06-88d2-0acb30a27c40\") " Sep 08 04:16:10.336314 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:10.336304 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqdkd\" (UniqueName: \"kubernetes.io/projected/a8f1e5c6-b9de-4b06-88d2-0acb30a27c40-kube-api-access-gqdkd\") pod \"a8f1e5c6-b9de-4b06-88d2-0acb30a27c40\" (UID: \"a8f1e5c6-b9de-4b06-88d2-0acb30a27c40\") " Sep 08 04:16:10.336839 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:10.336415 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ce41604-57e5-461a-b5d7-64deae23923f-plugins-config-volume\") pod \"7ce41604-57e5-461a-b5d7-64deae23923f\" (UID: \"7ce41604-57e5-461a-b5d7-64deae23923f\") " Sep 08 04:16:10.336839 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:10.336463 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cmkb\" (UniqueName: \"kubernetes.io/projected/7ce41604-57e5-461a-b5d7-64deae23923f-kube-api-access-4cmkb\") pod \"7ce41604-57e5-461a-b5d7-64deae23923f\" (UID: \"7ce41604-57e5-461a-b5d7-64deae23923f\") " Sep 08 04:16:10.336839 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:10.336758 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f1e5c6-b9de-4b06-88d2-0acb30a27c40-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a8f1e5c6-b9de-4b06-88d2-0acb30a27c40" (UID: "a8f1e5c6-b9de-4b06-88d2-0acb30a27c40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:16:10.336989 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:10.336830 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ce41604-57e5-461a-b5d7-64deae23923f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7ce41604-57e5-461a-b5d7-64deae23923f" (UID: "7ce41604-57e5-461a-b5d7-64deae23923f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:16:10.338830 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:10.338798 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f1e5c6-b9de-4b06-88d2-0acb30a27c40-kube-api-access-gqdkd" (OuterVolumeSpecName: "kube-api-access-gqdkd") pod "a8f1e5c6-b9de-4b06-88d2-0acb30a27c40" (UID: "a8f1e5c6-b9de-4b06-88d2-0acb30a27c40"). InnerVolumeSpecName "kube-api-access-gqdkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:16:10.339524 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:10.339475 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ce41604-57e5-461a-b5d7-64deae23923f-kube-api-access-4cmkb" (OuterVolumeSpecName: "kube-api-access-4cmkb") pod "7ce41604-57e5-461a-b5d7-64deae23923f" (UID: "7ce41604-57e5-461a-b5d7-64deae23923f"). InnerVolumeSpecName "kube-api-access-4cmkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:16:10.437994 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:10.437897 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cmkb\" (UniqueName: \"kubernetes.io/projected/7ce41604-57e5-461a-b5d7-64deae23923f-kube-api-access-4cmkb\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:16:10.437994 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:10.437929 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f1e5c6-b9de-4b06-88d2-0acb30a27c40-plugins-config-volume\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:16:10.437994 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:10.437940 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqdkd\" (UniqueName: \"kubernetes.io/projected/a8f1e5c6-b9de-4b06-88d2-0acb30a27c40-kube-api-access-gqdkd\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:16:10.437994 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:10.437950 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ce41604-57e5-461a-b5d7-64deae23923f-plugins-config-volume\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:16:10.567705 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:10.567671 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e0436e3-fc27-4639-b9d0-e988ff5bb8fa" path="/var/lib/kubelet/pods/3e0436e3-fc27-4639-b9d0-e988ff5bb8fa/volumes" Sep 08 04:16:10.894027 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:10.893989 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:10.993984 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:10.993941 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:11.091095 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:11.091066 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5" Sep 08 04:16:11.091252 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:11.091065 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5" event={"ID":"7ce41604-57e5-461a-b5d7-64deae23923f","Type":"ContainerDied","Data":"c2fa59ddd52f55df707c5802a7cc0716aa48e744b33c829d7a8c9e176f7c4df7"} Sep 08 04:16:11.091252 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:11.091218 2574 scope.go:117] "RemoveContainer" containerID="cec086022c17f55787310ceb5e791c9a421c3b70ec5e32a513df0027f1eb21e5" Sep 08 04:16:11.092725 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:11.092490 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x" Sep 08 04:16:11.092725 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:11.092533 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x" event={"ID":"a8f1e5c6-b9de-4b06-88d2-0acb30a27c40","Type":"ContainerDied","Data":"17f62156e5904c766d2cf91cb62673e7084735fe94c5b77bf13d210d7a38ecc2"} Sep 08 04:16:11.095896 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:11.095865 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:11.106798 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:11.106774 2574 scope.go:117] "RemoveContainer" containerID="19c6b9bb9506ee352922116566f1c3f8975d915ba92c0ae680e03cacc099d3e9" Sep 08 04:16:11.117185 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:11.117161 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x"] Sep 08 04:16:11.121322 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:11.121296 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-efa641f8-656c7554c7-ngj4x"] Sep 08 04:16:11.131488 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:11.131461 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5"] Sep 08 04:16:11.137633 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:11.137598 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-efa641f8-695fcf5ff8-lnxb5"] Sep 08 04:16:11.282478 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:11.282442 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:11.990310 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:11.990276 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-f0271e-29814015-v8wvj"] Sep 08 04:16:11.994340 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:11.994311 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:11.994918 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:11.994898 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-f0271e-29814015-v8wvj"] Sep 08 04:16:12.018670 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.018618 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f0271e-8558c55579-6jvbf"] Sep 08 04:16:12.018943 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.018910 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-f0271e-8558c55579-6jvbf" podUID="0384d0ca-874e-495a-9764-70043ed9bb83" containerName="payload-processing" containerID="cri-o://54d883776d9612322fb2a071e2bb7fff16ea31c0a25abecca86dc47b69e66f89" gracePeriod=30 Sep 08 04:16:12.028170 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.028131 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4"] Sep 08 04:16:12.028889 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.028377 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4" podUID="247f46a3-f13a-473a-9100-72fc5f426006" containerName="payload-pre-processing" containerID="cri-o://97b5c2c7bd740d67a242cceb79c5f10db13cdd437b5441c00dc0362837e7fb4d" gracePeriod=30 Sep 08 04:16:12.060702 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.060659 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:12.160468 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.160436 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:12.178598 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.178561 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0271e-lqbsh"] Sep 08 04:16:12.180923 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.180895 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f0271e-lqbsh"] Sep 08 04:16:12.217048 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.217021 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:12.302021 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.301999 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-f0271e-8558c55579-6jvbf" Sep 08 04:16:12.304817 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.304794 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4" Sep 08 04:16:12.357232 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.357183 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247f46a3-f13a-473a-9100-72fc5f426006-plugins-config-volume\") pod \"247f46a3-f13a-473a-9100-72fc5f426006\" (UID: \"247f46a3-f13a-473a-9100-72fc5f426006\") " Sep 08 04:16:12.357232 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.357236 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgl88\" (UniqueName: \"kubernetes.io/projected/247f46a3-f13a-473a-9100-72fc5f426006-kube-api-access-hgl88\") pod \"247f46a3-f13a-473a-9100-72fc5f426006\" (UID: \"247f46a3-f13a-473a-9100-72fc5f426006\") " Sep 08 04:16:12.357498 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.357329 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0384d0ca-874e-495a-9764-70043ed9bb83-plugins-config-volume\") pod \"0384d0ca-874e-495a-9764-70043ed9bb83\" (UID: \"0384d0ca-874e-495a-9764-70043ed9bb83\") " Sep 08 04:16:12.357498 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.357424 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chnwl\" (UniqueName: \"kubernetes.io/projected/0384d0ca-874e-495a-9764-70043ed9bb83-kube-api-access-chnwl\") pod \"0384d0ca-874e-495a-9764-70043ed9bb83\" (UID: \"0384d0ca-874e-495a-9764-70043ed9bb83\") " Sep 08 04:16:12.357734 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.357709 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0384d0ca-874e-495a-9764-70043ed9bb83-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0384d0ca-874e-495a-9764-70043ed9bb83" (UID: "0384d0ca-874e-495a-9764-70043ed9bb83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:16:12.357824 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.357809 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0384d0ca-874e-495a-9764-70043ed9bb83-plugins-config-volume\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:16:12.358126 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.358103 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247f46a3-f13a-473a-9100-72fc5f426006-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "247f46a3-f13a-473a-9100-72fc5f426006" (UID: "247f46a3-f13a-473a-9100-72fc5f426006"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:16:12.360048 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.360024 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247f46a3-f13a-473a-9100-72fc5f426006-kube-api-access-hgl88" (OuterVolumeSpecName: "kube-api-access-hgl88") pod "247f46a3-f13a-473a-9100-72fc5f426006" (UID: "247f46a3-f13a-473a-9100-72fc5f426006"). InnerVolumeSpecName "kube-api-access-hgl88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:16:12.360548 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.360521 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0384d0ca-874e-495a-9764-70043ed9bb83-kube-api-access-chnwl" (OuterVolumeSpecName: "kube-api-access-chnwl") pod "0384d0ca-874e-495a-9764-70043ed9bb83" (UID: "0384d0ca-874e-495a-9764-70043ed9bb83"). InnerVolumeSpecName "kube-api-access-chnwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:16:12.458729 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.458684 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247f46a3-f13a-473a-9100-72fc5f426006-plugins-config-volume\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:16:12.458729 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.458718 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgl88\" (UniqueName: \"kubernetes.io/projected/247f46a3-f13a-473a-9100-72fc5f426006-kube-api-access-hgl88\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:16:12.458991 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.458742 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chnwl\" (UniqueName: \"kubernetes.io/projected/0384d0ca-874e-495a-9764-70043ed9bb83-kube-api-access-chnwl\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:16:12.541432 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.541337 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:12.570992 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.570959 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36d3e936-1e88-4a73-a8c1-9fe9b2561531" path="/var/lib/kubelet/pods/36d3e936-1e88-4a73-a8c1-9fe9b2561531/volumes" Sep 08 04:16:12.571306 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.571293 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ce41604-57e5-461a-b5d7-64deae23923f" path="/var/lib/kubelet/pods/7ce41604-57e5-461a-b5d7-64deae23923f/volumes" Sep 08 04:16:12.571604 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.571592 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="924f5e79-95f6-4f4e-b2d3-ec38c905aa76" path="/var/lib/kubelet/pods/924f5e79-95f6-4f4e-b2d3-ec38c905aa76/volumes" Sep 08 04:16:12.571905 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:12.571894 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8f1e5c6-b9de-4b06-88d2-0acb30a27c40" path="/var/lib/kubelet/pods/a8f1e5c6-b9de-4b06-88d2-0acb30a27c40/volumes" Sep 08 04:16:13.109989 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:13.109954 2574 generic.go:356] "Generic (PLEG): container finished" podID="247f46a3-f13a-473a-9100-72fc5f426006" containerID="97b5c2c7bd740d67a242cceb79c5f10db13cdd437b5441c00dc0362837e7fb4d" exitCode=0 Sep 08 04:16:13.110429 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:13.109996 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4" Sep 08 04:16:13.110429 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:13.110041 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4" event={"ID":"247f46a3-f13a-473a-9100-72fc5f426006","Type":"ContainerDied","Data":"97b5c2c7bd740d67a242cceb79c5f10db13cdd437b5441c00dc0362837e7fb4d"} Sep 08 04:16:13.110429 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:13.110095 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4" event={"ID":"247f46a3-f13a-473a-9100-72fc5f426006","Type":"ContainerDied","Data":"2288216a6d1de1cb24b17970684e3a1966e614477bea5559e751426d86279805"} Sep 08 04:16:13.110429 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:13.110119 2574 scope.go:117] "RemoveContainer" containerID="97b5c2c7bd740d67a242cceb79c5f10db13cdd437b5441c00dc0362837e7fb4d" Sep 08 04:16:13.111423 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:13.111400 2574 generic.go:356] "Generic (PLEG): container finished" podID="0384d0ca-874e-495a-9764-70043ed9bb83" containerID="54d883776d9612322fb2a071e2bb7fff16ea31c0a25abecca86dc47b69e66f89" exitCode=0 Sep 08 04:16:13.111574 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:13.111441 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-f0271e-8558c55579-6jvbf" event={"ID":"0384d0ca-874e-495a-9764-70043ed9bb83","Type":"ContainerDied","Data":"54d883776d9612322fb2a071e2bb7fff16ea31c0a25abecca86dc47b69e66f89"} Sep 08 04:16:13.111574 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:13.111456 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-f0271e-8558c55579-6jvbf" Sep 08 04:16:13.111574 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:13.111483 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-f0271e-8558c55579-6jvbf" event={"ID":"0384d0ca-874e-495a-9764-70043ed9bb83","Type":"ContainerDied","Data":"74eb1b8e610989501588363fd5c121d57e5b2c27cacd8db250b20811671ccd41"} Sep 08 04:16:13.120135 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:13.120107 2574 scope.go:117] "RemoveContainer" containerID="97b5c2c7bd740d67a242cceb79c5f10db13cdd437b5441c00dc0362837e7fb4d" Sep 08 04:16:13.120457 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:16:13.120435 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97b5c2c7bd740d67a242cceb79c5f10db13cdd437b5441c00dc0362837e7fb4d\": container with ID starting with 97b5c2c7bd740d67a242cceb79c5f10db13cdd437b5441c00dc0362837e7fb4d not found: ID does not exist" containerID="97b5c2c7bd740d67a242cceb79c5f10db13cdd437b5441c00dc0362837e7fb4d" Sep 08 04:16:13.120554 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:13.120467 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97b5c2c7bd740d67a242cceb79c5f10db13cdd437b5441c00dc0362837e7fb4d"} err="failed to get container status \"97b5c2c7bd740d67a242cceb79c5f10db13cdd437b5441c00dc0362837e7fb4d\": rpc error: code = NotFound desc = could not find container \"97b5c2c7bd740d67a242cceb79c5f10db13cdd437b5441c00dc0362837e7fb4d\": container with ID starting with 97b5c2c7bd740d67a242cceb79c5f10db13cdd437b5441c00dc0362837e7fb4d not found: ID does not exist" Sep 08 04:16:13.120554 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:13.120484 2574 scope.go:117] "RemoveContainer" containerID="54d883776d9612322fb2a071e2bb7fff16ea31c0a25abecca86dc47b69e66f89" Sep 08 04:16:13.129544 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:13.129518 2574 scope.go:117] "RemoveContainer" containerID="54d883776d9612322fb2a071e2bb7fff16ea31c0a25abecca86dc47b69e66f89" Sep 08 04:16:13.129905 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:16:13.129883 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54d883776d9612322fb2a071e2bb7fff16ea31c0a25abecca86dc47b69e66f89\": container with ID starting with 54d883776d9612322fb2a071e2bb7fff16ea31c0a25abecca86dc47b69e66f89 not found: ID does not exist" containerID="54d883776d9612322fb2a071e2bb7fff16ea31c0a25abecca86dc47b69e66f89" Sep 08 04:16:13.129995 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:13.129911 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d883776d9612322fb2a071e2bb7fff16ea31c0a25abecca86dc47b69e66f89"} err="failed to get container status \"54d883776d9612322fb2a071e2bb7fff16ea31c0a25abecca86dc47b69e66f89\": rpc error: code = NotFound desc = could not find container \"54d883776d9612322fb2a071e2bb7fff16ea31c0a25abecca86dc47b69e66f89\": container with ID starting with 54d883776d9612322fb2a071e2bb7fff16ea31c0a25abecca86dc47b69e66f89 not found: ID does not exist" Sep 08 04:16:13.130286 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:13.130266 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f0271e-8558c55579-6jvbf"] Sep 08 04:16:13.133611 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:13.133587 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f0271e-8558c55579-6jvbf"] Sep 08 04:16:13.143757 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:13.143727 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4"] Sep 08 04:16:13.147965 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:13.147933 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f0271e-5785695f4b-rw7d4"] Sep 08 04:16:14.567650 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:14.567621 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0384d0ca-874e-495a-9764-70043ed9bb83" path="/var/lib/kubelet/pods/0384d0ca-874e-495a-9764-70043ed9bb83/volumes" Sep 08 04:16:14.568013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:14.568001 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="247f46a3-f13a-473a-9100-72fc5f426006" path="/var/lib/kubelet/pods/247f46a3-f13a-473a-9100-72fc5f426006/volumes" Sep 08 04:16:17.623422 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:17.623389 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:22.750544 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:22.750462 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:28.589726 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:28.589688 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:28.701007 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:28.700972 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:28.798705 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:28.798655 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:29.003154 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:29.003125 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:29.330658 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:29.330570 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:40.470901 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:40.470867 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:41.687209 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:41.687176 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:42.123421 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:42.123362 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:45.202892 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:45.202850 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:45.850403 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:45.850367 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:48.297213 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:48.297148 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:48.398150 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:48.398109 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:48.533600 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:48.533560 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp"] Sep 08 04:16:48.533841 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:48.533814 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" podUID="fb216dc0-7136-4cb5-b513-b8cf7ab9252c" containerName="istio-proxy" containerID="cri-o://dca3efd53ea5ed84c88ca192ded87e312d543931c9be757a1e7f423daa43df6a" gracePeriod=30 Sep 08 04:16:48.546697 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:48.546657 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:48.973615 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:48.973514 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:49.241926 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.241851 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962"] Sep 08 04:16:49.242241 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.242229 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="247f46a3-f13a-473a-9100-72fc5f426006" containerName="payload-pre-processing" Sep 08 04:16:49.242284 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.242245 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="247f46a3-f13a-473a-9100-72fc5f426006" containerName="payload-pre-processing" Sep 08 04:16:49.242284 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.242269 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8f1e5c6-b9de-4b06-88d2-0acb30a27c40" containerName="payload-processing" Sep 08 04:16:49.242284 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.242278 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8f1e5c6-b9de-4b06-88d2-0acb30a27c40" containerName="payload-processing" Sep 08 04:16:49.242407 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.242306 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0384d0ca-874e-495a-9764-70043ed9bb83" containerName="payload-processing" Sep 08 04:16:49.242407 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.242316 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0384d0ca-874e-495a-9764-70043ed9bb83" containerName="payload-processing" Sep 08 04:16:49.242407 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.242335 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ce41604-57e5-461a-b5d7-64deae23923f" containerName="payload-pre-processing" Sep 08 04:16:49.242407 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.242343 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ce41604-57e5-461a-b5d7-64deae23923f" containerName="payload-pre-processing" Sep 08 04:16:49.242407 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.242350 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="924f5e79-95f6-4f4e-b2d3-ec38c905aa76" containerName="revoke-keys" Sep 08 04:16:49.242407 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.242356 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="924f5e79-95f6-4f4e-b2d3-ec38c905aa76" containerName="revoke-keys" Sep 08 04:16:49.242407 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.242366 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e0436e3-fc27-4639-b9d0-e988ff5bb8fa" containerName="revoke-keys" Sep 08 04:16:49.242407 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.242371 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e0436e3-fc27-4639-b9d0-e988ff5bb8fa" containerName="revoke-keys" Sep 08 04:16:49.242715 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.242449 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8f1e5c6-b9de-4b06-88d2-0acb30a27c40" containerName="payload-processing" Sep 08 04:16:49.242715 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.242458 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ce41604-57e5-461a-b5d7-64deae23923f" containerName="payload-pre-processing" Sep 08 04:16:49.242715 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.242464 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="924f5e79-95f6-4f4e-b2d3-ec38c905aa76" containerName="revoke-keys" Sep 08 04:16:49.242715 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.242472 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0384d0ca-874e-495a-9764-70043ed9bb83" containerName="payload-processing" Sep 08 04:16:49.242715 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.242477 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="247f46a3-f13a-473a-9100-72fc5f426006" containerName="payload-pre-processing" Sep 08 04:16:49.242715 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.242484 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e0436e3-fc27-4639-b9d0-e988ff5bb8fa" containerName="revoke-keys" Sep 08 04:16:49.247460 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.247437 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.250185 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.250163 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-cb3759-openshift-default-dockercfg-fnmhq\"" Sep 08 04:16:49.259875 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.259845 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962"] Sep 08 04:16:49.272723 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.272657 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:49.369773 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.369738 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:49.387946 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.387911 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-istio-envoy\") pod \"e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.387946 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.387951 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d03860-2980-434f-a45b-d41427acc87c-istio-podinfo\") pod \"e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.388148 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.387974 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-credential-socket\") pod \"e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.388148 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.388059 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-istio-data\") pod \"e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.388148 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.388092 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-979dx\" (UniqueName: \"kubernetes.io/projected/00d03860-2980-434f-a45b-d41427acc87c-kube-api-access-979dx\") pod \"e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.388253 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.388164 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-workload-certs\") pod \"e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.388253 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.388185 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-workload-socket\") pod \"e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.388253 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.388199 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d03860-2980-434f-a45b-d41427acc87c-istiod-ca-cert\") pod \"e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.388253 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.388230 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d03860-2980-434f-a45b-d41427acc87c-istio-token\") pod \"e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.488698 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.488663 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-979dx\" (UniqueName: \"kubernetes.io/projected/00d03860-2980-434f-a45b-d41427acc87c-kube-api-access-979dx\") pod \"e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.488894 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.488728 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-workload-certs\") pod \"e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.488894 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.488750 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-workload-socket\") pod \"e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.488894 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.488767 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d03860-2980-434f-a45b-d41427acc87c-istiod-ca-cert\") pod \"e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.488894 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.488792 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d03860-2980-434f-a45b-d41427acc87c-istio-token\") pod \"e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.488894 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.488828 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-istio-envoy\") pod \"e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.488894 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.488857 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d03860-2980-434f-a45b-d41427acc87c-istio-podinfo\") pod \"e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.488894 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.488890 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-credential-socket\") pod \"e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.489277 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.488926 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-istio-data\") pod \"e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.489277 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.489183 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-workload-certs\") pod \"e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.489277 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.489264 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-credential-socket\") pod \"e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.489442 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.489321 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-istio-data\") pod \"e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.489648 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.489621 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d03860-2980-434f-a45b-d41427acc87c-istiod-ca-cert\") pod \"e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.489795 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.489653 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-workload-socket\") pod \"e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.491144 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.491122 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-istio-envoy\") pod \"e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.491421 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.491402 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d03860-2980-434f-a45b-d41427acc87c-istio-podinfo\") pod \"e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.499663 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.499635 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d03860-2980-434f-a45b-d41427acc87c-istio-token\") pod \"e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.499983 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.499964 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-979dx\" (UniqueName: \"kubernetes.io/projected/00d03860-2980-434f-a45b-d41427acc87c-kube-api-access-979dx\") pod \"e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.558958 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.558920 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:49.834381 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.834351 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962"] Sep 08 04:16:49.835844 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:16:49.835811 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d03860_2980_434f_a45b_d41427acc87c.slice/crio-f47aed1d1c8b702d5f7b67fea2660e2f7b52998bd0ed3652c12658e4d160d1e1 WatchSource:0}: Error finding container f47aed1d1c8b702d5f7b67fea2660e2f7b52998bd0ed3652c12658e4d160d1e1: Status 404 returned error can't find the container with id f47aed1d1c8b702d5f7b67fea2660e2f7b52998bd0ed3652c12658e4d160d1e1 Sep 08 04:16:49.837968 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.837938 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:16:49.838028 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.838004 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:16:49.838068 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:49.838031 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:16:50.334051 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:50.334003 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" event={"ID":"00d03860-2980-434f-a45b-d41427acc87c","Type":"ContainerStarted","Data":"c992d29a8708f537979591350b749c37b875038a3640d9bb00c8e257864ac738"} Sep 08 04:16:50.334279 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:50.334103 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" event={"ID":"00d03860-2980-434f-a45b-d41427acc87c","Type":"ContainerStarted","Data":"f47aed1d1c8b702d5f7b67fea2660e2f7b52998bd0ed3652c12658e4d160d1e1"} Sep 08 04:16:50.355547 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:50.355472 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" podStartSLOduration=1.355456871 podStartE2EDuration="1.355456871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:16:50.353565311 +0000 UTC m=+984.276026063" watchObservedRunningTime="2026-09-08 04:16:50.355456871 +0000 UTC m=+984.277917611" Sep 08 04:16:50.559624 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:50.559585 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:50.568207 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:50.568175 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:51.339799 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:51.339766 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:51.340732 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:51.340710 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:16:52.201589 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:52.201551 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:52.301027 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:52.300993 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:53.371770 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:53.371740 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:53.436398 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:53.436364 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:53.579934 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:53.579903 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:53.789393 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:53.789361 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:16:53.933409 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:53.933322 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istio-token\") pod \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " Sep 08 04:16:53.933409 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:53.933367 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istio-podinfo\") pod \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " Sep 08 04:16:53.933409 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:53.933407 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-workload-socket\") pod \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " Sep 08 04:16:53.933718 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:53.933423 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-credential-socket\") pod \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " Sep 08 04:16:53.933718 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:53.933443 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kms6f\" (UniqueName: \"kubernetes.io/projected/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-kube-api-access-kms6f\") pod \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " Sep 08 04:16:53.933718 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:53.933484 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istio-envoy\") pod \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " Sep 08 04:16:53.933718 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:53.933551 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istiod-ca-cert\") pod \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " Sep 08 04:16:53.933718 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:53.933601 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-workload-certs\") pod \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " Sep 08 04:16:53.933718 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:53.933623 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istio-data\") pod \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\" (UID: \"fb216dc0-7136-4cb5-b513-b8cf7ab9252c\") " Sep 08 04:16:53.934023 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:53.933750 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fb216dc0-7136-4cb5-b513-b8cf7ab9252c" (UID: "fb216dc0-7136-4cb5-b513-b8cf7ab9252c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:16:53.934023 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:53.933972 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istio-data" (OuterVolumeSpecName: "istio-data") pod "fb216dc0-7136-4cb5-b513-b8cf7ab9252c" (UID: "fb216dc0-7136-4cb5-b513-b8cf7ab9252c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:16:53.934023 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:53.933764 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fb216dc0-7136-4cb5-b513-b8cf7ab9252c" (UID: "fb216dc0-7136-4cb5-b513-b8cf7ab9252c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:16:53.934023 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:53.933993 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fb216dc0-7136-4cb5-b513-b8cf7ab9252c" (UID: "fb216dc0-7136-4cb5-b513-b8cf7ab9252c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:16:53.934233 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:53.934029 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-workload-socket\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:16:53.934233 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:53.934075 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fb216dc0-7136-4cb5-b513-b8cf7ab9252c" (UID: "fb216dc0-7136-4cb5-b513-b8cf7ab9252c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:16:53.935809 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:53.935782 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istio-token" (OuterVolumeSpecName: "istio-token") pod "fb216dc0-7136-4cb5-b513-b8cf7ab9252c" (UID: "fb216dc0-7136-4cb5-b513-b8cf7ab9252c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:16:53.935932 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:53.935820 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-kube-api-access-kms6f" (OuterVolumeSpecName: "kube-api-access-kms6f") pod "fb216dc0-7136-4cb5-b513-b8cf7ab9252c" (UID: "fb216dc0-7136-4cb5-b513-b8cf7ab9252c"). InnerVolumeSpecName "kube-api-access-kms6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:16:53.935932 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:53.935914 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fb216dc0-7136-4cb5-b513-b8cf7ab9252c" (UID: "fb216dc0-7136-4cb5-b513-b8cf7ab9252c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:16:53.936140 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:53.936113 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fb216dc0-7136-4cb5-b513-b8cf7ab9252c" (UID: "fb216dc0-7136-4cb5-b513-b8cf7ab9252c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 04:16:53.952032 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:53.952000 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:54.035365 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.035327 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kms6f\" (UniqueName: \"kubernetes.io/projected/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-kube-api-access-kms6f\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:16:54.035365 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.035362 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istio-envoy\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:16:54.035365 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.035372 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istiod-ca-cert\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:16:54.035730 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.035380 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-workload-certs\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:16:54.035730 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.035389 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istio-data\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:16:54.035730 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.035396 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istio-token\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:16:54.035730 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.035403 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-istio-podinfo\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:16:54.035730 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.035411 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb216dc0-7136-4cb5-b513-b8cf7ab9252c-credential-socket\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:16:54.042435 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.042408 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:54.050616 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.050574 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts"] Sep 08 04:16:54.050996 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.050976 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb216dc0-7136-4cb5-b513-b8cf7ab9252c" containerName="istio-proxy" Sep 08 04:16:54.051084 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.051001 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb216dc0-7136-4cb5-b513-b8cf7ab9252c" containerName="istio-proxy" Sep 08 04:16:54.051084 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.051017 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb216dc0-7136-4cb5-b513-b8cf7ab9252c" containerName="istio-proxy" Sep 08 04:16:54.051084 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.051025 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb216dc0-7136-4cb5-b513-b8cf7ab9252c" containerName="istio-proxy" Sep 08 04:16:54.051236 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.051135 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb216dc0-7136-4cb5-b513-b8cf7ab9252c" containerName="istio-proxy" Sep 08 04:16:54.051236 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.051150 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb216dc0-7136-4cb5-b513-b8cf7ab9252c" containerName="istio-proxy" Sep 08 04:16:54.054442 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.054422 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts" Sep 08 04:16:54.056775 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.056740 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-cb3759\"" Sep 08 04:16:54.056899 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.056856 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-cb3759-dockercfg-rrqmb\"" Sep 08 04:16:54.064629 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.064598 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts"] Sep 08 04:16:54.084610 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.084574 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq"] Sep 08 04:16:54.087959 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.087935 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq" Sep 08 04:16:54.098778 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.098750 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq"] Sep 08 04:16:54.237794 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.237695 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdcs4\" (UniqueName: \"kubernetes.io/projected/862c4066-bdde-4949-a1f3-c078d0ac9da0-kube-api-access-rdcs4\") pod \"payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts\" (UID: \"862c4066-bdde-4949-a1f3-c078d0ac9da0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts" Sep 08 04:16:54.237794 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.237744 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f400055-8987-42d5-a021-0b48600e6c8b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq\" (UID: \"1f400055-8987-42d5-a021-0b48600e6c8b\") " pod="openshift-ingress/payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq" Sep 08 04:16:54.237995 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.237896 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5b92\" (UniqueName: \"kubernetes.io/projected/1f400055-8987-42d5-a021-0b48600e6c8b-kube-api-access-z5b92\") pod \"payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq\" (UID: \"1f400055-8987-42d5-a021-0b48600e6c8b\") " pod="openshift-ingress/payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq" Sep 08 04:16:54.237995 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.237951 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862c4066-bdde-4949-a1f3-c078d0ac9da0-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts\" (UID: \"862c4066-bdde-4949-a1f3-c078d0ac9da0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts" Sep 08 04:16:54.338789 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.338745 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f400055-8987-42d5-a021-0b48600e6c8b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq\" (UID: \"1f400055-8987-42d5-a021-0b48600e6c8b\") " pod="openshift-ingress/payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq" Sep 08 04:16:54.338970 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.338873 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5b92\" (UniqueName: \"kubernetes.io/projected/1f400055-8987-42d5-a021-0b48600e6c8b-kube-api-access-z5b92\") pod \"payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq\" (UID: \"1f400055-8987-42d5-a021-0b48600e6c8b\") " pod="openshift-ingress/payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq" Sep 08 04:16:54.338970 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.338933 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862c4066-bdde-4949-a1f3-c078d0ac9da0-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts\" (UID: \"862c4066-bdde-4949-a1f3-c078d0ac9da0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts" Sep 08 04:16:54.339124 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.339100 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdcs4\" (UniqueName: \"kubernetes.io/projected/862c4066-bdde-4949-a1f3-c078d0ac9da0-kube-api-access-rdcs4\") pod \"payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts\" (UID: \"862c4066-bdde-4949-a1f3-c078d0ac9da0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts" Sep 08 04:16:54.339772 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.339751 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862c4066-bdde-4949-a1f3-c078d0ac9da0-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts\" (UID: \"862c4066-bdde-4949-a1f3-c078d0ac9da0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts" Sep 08 04:16:54.339829 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.339751 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f400055-8987-42d5-a021-0b48600e6c8b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq\" (UID: \"1f400055-8987-42d5-a021-0b48600e6c8b\") " pod="openshift-ingress/payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq" Sep 08 04:16:54.351271 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.351235 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5b92\" (UniqueName: \"kubernetes.io/projected/1f400055-8987-42d5-a021-0b48600e6c8b-kube-api-access-z5b92\") pod \"payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq\" (UID: \"1f400055-8987-42d5-a021-0b48600e6c8b\") " pod="openshift-ingress/payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq" Sep 08 04:16:54.352351 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.352330 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdcs4\" (UniqueName: \"kubernetes.io/projected/862c4066-bdde-4949-a1f3-c078d0ac9da0-kube-api-access-rdcs4\") pod \"payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts\" (UID: \"862c4066-bdde-4949-a1f3-c078d0ac9da0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts" Sep 08 04:16:54.365563 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.365525 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts" Sep 08 04:16:54.370473 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.370443 2574 generic.go:356] "Generic (PLEG): container finished" podID="fb216dc0-7136-4cb5-b513-b8cf7ab9252c" containerID="dca3efd53ea5ed84c88ca192ded87e312d543931c9be757a1e7f423daa43df6a" exitCode=0 Sep 08 04:16:54.370618 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.370538 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" event={"ID":"fb216dc0-7136-4cb5-b513-b8cf7ab9252c","Type":"ContainerDied","Data":"dca3efd53ea5ed84c88ca192ded87e312d543931c9be757a1e7f423daa43df6a"} Sep 08 04:16:54.370618 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.370600 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" Sep 08 04:16:54.370705 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.370634 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp" event={"ID":"fb216dc0-7136-4cb5-b513-b8cf7ab9252c","Type":"ContainerDied","Data":"c5c8f79a19d59bd987b002da54d28c09c3dcfa059872309204d4d79845970ae3"} Sep 08 04:16:54.370705 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.370668 2574 scope.go:117] "RemoveContainer" containerID="dca3efd53ea5ed84c88ca192ded87e312d543931c9be757a1e7f423daa43df6a" Sep 08 04:16:54.385532 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.385483 2574 scope.go:117] "RemoveContainer" containerID="a14b5019a938da481136fd6338eee6167e23ff01fad56d758b9137081107a6ce" Sep 08 04:16:54.395439 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.395403 2574 scope.go:117] "RemoveContainer" containerID="dca3efd53ea5ed84c88ca192ded87e312d543931c9be757a1e7f423daa43df6a" Sep 08 04:16:54.395865 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:16:54.395837 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca3efd53ea5ed84c88ca192ded87e312d543931c9be757a1e7f423daa43df6a\": container with ID starting with dca3efd53ea5ed84c88ca192ded87e312d543931c9be757a1e7f423daa43df6a not found: ID does not exist" containerID="dca3efd53ea5ed84c88ca192ded87e312d543931c9be757a1e7f423daa43df6a" Sep 08 04:16:54.395980 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.395877 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca3efd53ea5ed84c88ca192ded87e312d543931c9be757a1e7f423daa43df6a"} err="failed to get container status \"dca3efd53ea5ed84c88ca192ded87e312d543931c9be757a1e7f423daa43df6a\": rpc error: code = NotFound desc = could not find container \"dca3efd53ea5ed84c88ca192ded87e312d543931c9be757a1e7f423daa43df6a\": container with ID starting with dca3efd53ea5ed84c88ca192ded87e312d543931c9be757a1e7f423daa43df6a not found: ID does not exist" Sep 08 04:16:54.395980 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.395904 2574 scope.go:117] "RemoveContainer" containerID="a14b5019a938da481136fd6338eee6167e23ff01fad56d758b9137081107a6ce" Sep 08 04:16:54.396264 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:16:54.396235 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a14b5019a938da481136fd6338eee6167e23ff01fad56d758b9137081107a6ce\": container with ID starting with a14b5019a938da481136fd6338eee6167e23ff01fad56d758b9137081107a6ce not found: ID does not exist" containerID="a14b5019a938da481136fd6338eee6167e23ff01fad56d758b9137081107a6ce" Sep 08 04:16:54.396317 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.396267 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a14b5019a938da481136fd6338eee6167e23ff01fad56d758b9137081107a6ce"} err="failed to get container status \"a14b5019a938da481136fd6338eee6167e23ff01fad56d758b9137081107a6ce\": rpc error: code = NotFound desc = could not find container \"a14b5019a938da481136fd6338eee6167e23ff01fad56d758b9137081107a6ce\": container with ID starting with a14b5019a938da481136fd6338eee6167e23ff01fad56d758b9137081107a6ce not found: ID does not exist" Sep 08 04:16:54.398627 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.398605 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq" Sep 08 04:16:54.400468 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.400441 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp"] Sep 08 04:16:54.404434 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.404409 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-f0271e-openshift-default-598dd6b97b-2qjdp"] Sep 08 04:16:54.568410 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.568378 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb216dc0-7136-4cb5-b513-b8cf7ab9252c" path="/var/lib/kubelet/pods/fb216dc0-7136-4cb5-b513-b8cf7ab9252c/volumes" Sep 08 04:16:54.645420 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.645395 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts"] Sep 08 04:16:54.647034 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:16:54.647002 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod862c4066_bdde_4949_a1f3_c078d0ac9da0.slice/crio-6f65697f413c3afc8eb53fcbd0cfec3ff2bca14a08e274581dd50e751582ca0c WatchSource:0}: Error finding container 6f65697f413c3afc8eb53fcbd0cfec3ff2bca14a08e274581dd50e751582ca0c: Status 404 returned error can't find the container with id 6f65697f413c3afc8eb53fcbd0cfec3ff2bca14a08e274581dd50e751582ca0c Sep 08 04:16:54.687535 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:54.687491 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq"] Sep 08 04:16:54.688045 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:16:54.688013 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f400055_8987_42d5_a021_0b48600e6c8b.slice/crio-224e387a7062f9a844e085b02af6d16d9213cf5d15c62539881ee8d0d232c4a2 WatchSource:0}: Error finding container 224e387a7062f9a844e085b02af6d16d9213cf5d15c62539881ee8d0d232c4a2: Status 404 returned error can't find the container with id 224e387a7062f9a844e085b02af6d16d9213cf5d15c62539881ee8d0d232c4a2 Sep 08 04:16:55.210357 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:55.210324 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:16:55.380500 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:55.380463 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq" event={"ID":"1f400055-8987-42d5-a021-0b48600e6c8b","Type":"ContainerStarted","Data":"07f68f71daeb7e9d6dcaaabed7c66e97b145f1770afcb01546b7c58afac9a314"} Sep 08 04:16:55.380748 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:55.380543 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq" event={"ID":"1f400055-8987-42d5-a021-0b48600e6c8b","Type":"ContainerStarted","Data":"224e387a7062f9a844e085b02af6d16d9213cf5d15c62539881ee8d0d232c4a2"} Sep 08 04:16:55.380826 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:55.380760 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq" Sep 08 04:16:55.382818 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:55.382786 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts" event={"ID":"862c4066-bdde-4949-a1f3-c078d0ac9da0","Type":"ContainerStarted","Data":"59416758e499c8e902d89dad6840f360e8c65c02514dfe3af1e8d63b327df89f"} Sep 08 04:16:55.382936 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:55.382833 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts" event={"ID":"862c4066-bdde-4949-a1f3-c078d0ac9da0","Type":"ContainerStarted","Data":"6f65697f413c3afc8eb53fcbd0cfec3ff2bca14a08e274581dd50e751582ca0c"} Sep 08 04:16:55.382936 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:55.382877 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts" Sep 08 04:16:55.397529 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:16:55.396182 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq" podStartSLOduration=1.3961637150000001 podStartE2EDuration="1.396163715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:16:55.395775465 +0000 UTC m=+989.318236231" watchObservedRunningTime="2026-09-08 04:16:55.396163715 +0000 UTC m=+989.318624459" Sep 08 04:17:01.374496 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:01.374458 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:01.545140 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:01.545106 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:01.638010 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:01.637917 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:04.687531 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:04.687487 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:06.392158 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:06.392124 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq" Sep 08 04:17:06.393740 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:06.393713 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts" Sep 08 04:17:06.412987 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:06.412915 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts" podStartSLOduration=12.412892002 podStartE2EDuration="12.412892002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:16:55.416187112 +0000 UTC m=+989.338647866" watchObservedRunningTime="2026-09-08 04:17:06.412892002 +0000 UTC m=+1000.335352744" Sep 08 04:17:12.493201 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:12.493168 2574 generic.go:356] "Generic (PLEG): container finished" podID="00d03860-2980-434f-a45b-d41427acc87c" containerID="c992d29a8708f537979591350b749c37b875038a3640d9bb00c8e257864ac738" exitCode=0 Sep 08 04:17:12.493660 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:12.493226 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" event={"ID":"00d03860-2980-434f-a45b-d41427acc87c","Type":"ContainerDied","Data":"c992d29a8708f537979591350b749c37b875038a3640d9bb00c8e257864ac738"} Sep 08 04:17:12.493717 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:12.493688 2574 scope.go:117] "RemoveContainer" containerID="c992d29a8708f537979591350b749c37b875038a3640d9bb00c8e257864ac738" Sep 08 04:17:12.495094 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:12.495067 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:17:12.495271 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:12.495251 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:17:12.495397 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:12.495374 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:17:13.505809 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:13.505759 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" event={"ID":"00d03860-2980-434f-a45b-d41427acc87c","Type":"ContainerStarted","Data":"c8fa18c6bb66d9033672de931f4cd4d038258d3b5368356ec7ceb4ccafacae30"} Sep 08 04:17:13.559428 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:13.559390 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:17:13.559639 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:13.559620 2574 patch_prober.go:28] interesting pod/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Sep 08 04:17:13.559725 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:13.559674 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" podUID="00d03860-2980-434f-a45b-d41427acc87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Sep 08 04:17:14.559960 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:14.559927 2574 patch_prober.go:28] interesting pod/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Sep 08 04:17:14.560331 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:14.559989 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" podUID="00d03860-2980-434f-a45b-d41427acc87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Sep 08 04:17:15.559756 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:15.559721 2574 patch_prober.go:28] interesting pod/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Sep 08 04:17:15.559932 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:15.559774 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" podUID="00d03860-2980-434f-a45b-d41427acc87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Sep 08 04:17:16.559462 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:16.559428 2574 patch_prober.go:28] interesting pod/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Sep 08 04:17:16.559842 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:16.559521 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" podUID="00d03860-2980-434f-a45b-d41427acc87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Sep 08 04:17:16.575909 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:16.575861 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:17.126088 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:17.126052 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:17.371427 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:17.371385 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:17.560153 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:17.560120 2574 patch_prober.go:28] interesting pod/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Sep 08 04:17:17.560559 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:17.560173 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" podUID="00d03860-2980-434f-a45b-d41427acc87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Sep 08 04:17:18.560357 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:18.560309 2574 patch_prober.go:28] interesting pod/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Sep 08 04:17:18.560802 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:18.560385 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" podUID="00d03860-2980-434f-a45b-d41427acc87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Sep 08 04:17:19.562974 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:19.562946 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:17:19.563446 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:19.563429 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:17:19.564030 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:19.564011 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:17:23.295753 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:23.295721 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:23.397205 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:23.397176 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:23.496591 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:23.496564 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:33.981652 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:33.981613 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:34.717278 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:34.717214 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:38.176358 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:38.176323 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:40.882220 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:40.882187 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:40.983566 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:40.983531 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:41.083210 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:41.083178 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:41.182915 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:41.182825 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:41.823299 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:41.823260 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:41.904657 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:41.904621 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:43.374683 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:43.374612 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:47.241380 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:47.241347 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:47.342615 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:47.342560 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb3759-r9r8q"] Sep 08 04:17:47.347320 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:47.347301 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb3759-r9r8q" Sep 08 04:17:47.351147 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:47.351120 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7r29w\"" Sep 08 04:17:47.351433 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:47.351124 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 04:17:47.354061 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:47.354034 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb3759-r9r8q"] Sep 08 04:17:47.523353 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:47.523317 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf04b0d-fb46-413b-8312-82bf76b93186-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-cb3759-r9r8q\" (UID: \"2bf04b0d-fb46-413b-8312-82bf76b93186\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb3759-r9r8q" Sep 08 04:17:47.624073 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:47.624035 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf04b0d-fb46-413b-8312-82bf76b93186-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-cb3759-r9r8q\" (UID: \"2bf04b0d-fb46-413b-8312-82bf76b93186\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb3759-r9r8q" Sep 08 04:17:47.624706 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:47.624686 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf04b0d-fb46-413b-8312-82bf76b93186-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-cb3759-r9r8q\" (UID: \"2bf04b0d-fb46-413b-8312-82bf76b93186\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb3759-r9r8q" Sep 08 04:17:47.660177 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:47.660133 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb3759-r9r8q" Sep 08 04:17:47.933737 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:47.933673 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb3759-r9r8q"] Sep 08 04:17:47.935968 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:17:47.935941 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf04b0d_fb46_413b_8312_82bf76b93186.slice/crio-7153a4609cddd586d7e8c4f15233ea4351fbf9fdb38817a5fce5a96d75a88470 WatchSource:0}: Error finding container 7153a4609cddd586d7e8c4f15233ea4351fbf9fdb38817a5fce5a96d75a88470: Status 404 returned error can't find the container with id 7153a4609cddd586d7e8c4f15233ea4351fbf9fdb38817a5fce5a96d75a88470 Sep 08 04:17:48.719886 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:48.719855 2574 generic.go:356] "Generic (PLEG): container finished" podID="2bf04b0d-fb46-413b-8312-82bf76b93186" containerID="e5c1dc7f49c52797c5e39e5949b5be2b0c99a9650adc3ef04731b63f7144f90d" exitCode=0 Sep 08 04:17:48.720246 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:48.719915 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb3759-r9r8q" event={"ID":"2bf04b0d-fb46-413b-8312-82bf76b93186","Type":"ContainerDied","Data":"e5c1dc7f49c52797c5e39e5949b5be2b0c99a9650adc3ef04731b63f7144f90d"} Sep 08 04:17:48.720246 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:48.719957 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb3759-r9r8q" event={"ID":"2bf04b0d-fb46-413b-8312-82bf76b93186","Type":"ContainerStarted","Data":"7153a4609cddd586d7e8c4f15233ea4351fbf9fdb38817a5fce5a96d75a88470"} Sep 08 04:17:49.340682 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:49.340647 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:49.399306 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:49.399274 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:49.858317 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:49.858293 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb3759-r9r8q" Sep 08 04:17:49.945536 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:49.945471 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf04b0d-fb46-413b-8312-82bf76b93186-maas-api-service-ca\") pod \"2bf04b0d-fb46-413b-8312-82bf76b93186\" (UID: \"2bf04b0d-fb46-413b-8312-82bf76b93186\") " Sep 08 04:17:49.945853 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:49.945829 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf04b0d-fb46-413b-8312-82bf76b93186-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2bf04b0d-fb46-413b-8312-82bf76b93186" (UID: "2bf04b0d-fb46-413b-8312-82bf76b93186"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:17:49.990369 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:49.990290 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:50.046458 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:50.046426 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf04b0d-fb46-413b-8312-82bf76b93186-maas-api-service-ca\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:17:50.732754 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:50.732724 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb3759-r9r8q" Sep 08 04:17:50.732910 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:50.732764 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb3759-r9r8q" event={"ID":"2bf04b0d-fb46-413b-8312-82bf76b93186","Type":"ContainerDied","Data":"7153a4609cddd586d7e8c4f15233ea4351fbf9fdb38817a5fce5a96d75a88470"} Sep 08 04:17:50.732910 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:50.732811 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7153a4609cddd586d7e8c4f15233ea4351fbf9fdb38817a5fce5a96d75a88470" Sep 08 04:17:56.173719 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:56.173688 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:56.289514 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:56.289461 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:56.419275 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:56.419244 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:56.728135 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:56.728099 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:56.862554 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:56.862492 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:58.152068 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.152034 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:58.187870 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.187834 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq"] Sep 08 04:17:58.188127 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.188091 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq" podUID="1f400055-8987-42d5-a021-0b48600e6c8b" containerName="payload-processing" containerID="cri-o://07f68f71daeb7e9d6dcaaabed7c66e97b145f1770afcb01546b7c58afac9a314" gracePeriod=30 Sep 08 04:17:58.200039 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.200011 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts"] Sep 08 04:17:58.200210 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.200190 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts" podUID="862c4066-bdde-4949-a1f3-c078d0ac9da0" containerName="payload-pre-processing" containerID="cri-o://59416758e499c8e902d89dad6840f360e8c65c02514dfe3af1e8d63b327df89f" gracePeriod=30 Sep 08 04:17:58.258727 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.258693 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:58.364475 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.364388 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:58.469820 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.469789 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb3759-r9r8q"] Sep 08 04:17:58.473675 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.473644 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb3759-r9r8q"] Sep 08 04:17:58.481377 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.481355 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:58.489350 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.489328 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq" Sep 08 04:17:58.492221 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.492203 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts" Sep 08 04:17:58.526478 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.526446 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5b92\" (UniqueName: \"kubernetes.io/projected/1f400055-8987-42d5-a021-0b48600e6c8b-kube-api-access-z5b92\") pod \"1f400055-8987-42d5-a021-0b48600e6c8b\" (UID: \"1f400055-8987-42d5-a021-0b48600e6c8b\") " Sep 08 04:17:58.526478 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.526482 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862c4066-bdde-4949-a1f3-c078d0ac9da0-plugins-config-volume\") pod \"862c4066-bdde-4949-a1f3-c078d0ac9da0\" (UID: \"862c4066-bdde-4949-a1f3-c078d0ac9da0\") " Sep 08 04:17:58.526709 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.526528 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f400055-8987-42d5-a021-0b48600e6c8b-plugins-config-volume\") pod \"1f400055-8987-42d5-a021-0b48600e6c8b\" (UID: \"1f400055-8987-42d5-a021-0b48600e6c8b\") " Sep 08 04:17:58.526709 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.526638 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdcs4\" (UniqueName: \"kubernetes.io/projected/862c4066-bdde-4949-a1f3-c078d0ac9da0-kube-api-access-rdcs4\") pod \"862c4066-bdde-4949-a1f3-c078d0ac9da0\" (UID: \"862c4066-bdde-4949-a1f3-c078d0ac9da0\") " Sep 08 04:17:58.526904 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.526876 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f400055-8987-42d5-a021-0b48600e6c8b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1f400055-8987-42d5-a021-0b48600e6c8b" (UID: "1f400055-8987-42d5-a021-0b48600e6c8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:17:58.526966 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.526902 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/862c4066-bdde-4949-a1f3-c078d0ac9da0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "862c4066-bdde-4949-a1f3-c078d0ac9da0" (UID: "862c4066-bdde-4949-a1f3-c078d0ac9da0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:17:58.528764 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.528736 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/862c4066-bdde-4949-a1f3-c078d0ac9da0-kube-api-access-rdcs4" (OuterVolumeSpecName: "kube-api-access-rdcs4") pod "862c4066-bdde-4949-a1f3-c078d0ac9da0" (UID: "862c4066-bdde-4949-a1f3-c078d0ac9da0"). InnerVolumeSpecName "kube-api-access-rdcs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:17:58.528929 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.528786 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f400055-8987-42d5-a021-0b48600e6c8b-kube-api-access-z5b92" (OuterVolumeSpecName: "kube-api-access-z5b92") pod "1f400055-8987-42d5-a021-0b48600e6c8b" (UID: "1f400055-8987-42d5-a021-0b48600e6c8b"). InnerVolumeSpecName "kube-api-access-z5b92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:17:58.573489 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.573455 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bf04b0d-fb46-413b-8312-82bf76b93186" path="/var/lib/kubelet/pods/2bf04b0d-fb46-413b-8312-82bf76b93186/volumes" Sep 08 04:17:58.628609 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.628573 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdcs4\" (UniqueName: \"kubernetes.io/projected/862c4066-bdde-4949-a1f3-c078d0ac9da0-kube-api-access-rdcs4\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:17:58.628609 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.628608 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5b92\" (UniqueName: \"kubernetes.io/projected/1f400055-8987-42d5-a021-0b48600e6c8b-kube-api-access-z5b92\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:17:58.628888 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.628620 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862c4066-bdde-4949-a1f3-c078d0ac9da0-plugins-config-volume\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:17:58.628888 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.628632 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f400055-8987-42d5-a021-0b48600e6c8b-plugins-config-volume\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:17:58.779675 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.779639 2574 generic.go:356] "Generic (PLEG): container finished" podID="1f400055-8987-42d5-a021-0b48600e6c8b" containerID="07f68f71daeb7e9d6dcaaabed7c66e97b145f1770afcb01546b7c58afac9a314" exitCode=0 Sep 08 04:17:58.779866 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.779684 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq" Sep 08 04:17:58.779866 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.779728 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq" event={"ID":"1f400055-8987-42d5-a021-0b48600e6c8b","Type":"ContainerDied","Data":"07f68f71daeb7e9d6dcaaabed7c66e97b145f1770afcb01546b7c58afac9a314"} Sep 08 04:17:58.779866 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.779793 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq" event={"ID":"1f400055-8987-42d5-a021-0b48600e6c8b","Type":"ContainerDied","Data":"224e387a7062f9a844e085b02af6d16d9213cf5d15c62539881ee8d0d232c4a2"} Sep 08 04:17:58.779866 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.779825 2574 scope.go:117] "RemoveContainer" containerID="07f68f71daeb7e9d6dcaaabed7c66e97b145f1770afcb01546b7c58afac9a314" Sep 08 04:17:58.781264 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.781242 2574 generic.go:356] "Generic (PLEG): container finished" podID="862c4066-bdde-4949-a1f3-c078d0ac9da0" containerID="59416758e499c8e902d89dad6840f360e8c65c02514dfe3af1e8d63b327df89f" exitCode=0 Sep 08 04:17:58.781360 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.781280 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts" Sep 08 04:17:58.781360 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.781339 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts" event={"ID":"862c4066-bdde-4949-a1f3-c078d0ac9da0","Type":"ContainerDied","Data":"59416758e499c8e902d89dad6840f360e8c65c02514dfe3af1e8d63b327df89f"} Sep 08 04:17:58.781471 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.781389 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts" event={"ID":"862c4066-bdde-4949-a1f3-c078d0ac9da0","Type":"ContainerDied","Data":"6f65697f413c3afc8eb53fcbd0cfec3ff2bca14a08e274581dd50e751582ca0c"} Sep 08 04:17:58.789445 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.789429 2574 scope.go:117] "RemoveContainer" containerID="07f68f71daeb7e9d6dcaaabed7c66e97b145f1770afcb01546b7c58afac9a314" Sep 08 04:17:58.789718 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:17:58.789698 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07f68f71daeb7e9d6dcaaabed7c66e97b145f1770afcb01546b7c58afac9a314\": container with ID starting with 07f68f71daeb7e9d6dcaaabed7c66e97b145f1770afcb01546b7c58afac9a314 not found: ID does not exist" containerID="07f68f71daeb7e9d6dcaaabed7c66e97b145f1770afcb01546b7c58afac9a314" Sep 08 04:17:58.789783 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.789725 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07f68f71daeb7e9d6dcaaabed7c66e97b145f1770afcb01546b7c58afac9a314"} err="failed to get container status \"07f68f71daeb7e9d6dcaaabed7c66e97b145f1770afcb01546b7c58afac9a314\": rpc error: code = NotFound desc = could not find container \"07f68f71daeb7e9d6dcaaabed7c66e97b145f1770afcb01546b7c58afac9a314\": container with ID starting with 07f68f71daeb7e9d6dcaaabed7c66e97b145f1770afcb01546b7c58afac9a314 not found: ID does not exist" Sep 08 04:17:58.789783 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.789741 2574 scope.go:117] "RemoveContainer" containerID="59416758e499c8e902d89dad6840f360e8c65c02514dfe3af1e8d63b327df89f" Sep 08 04:17:58.798048 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.798026 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:17:58.798532 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.798515 2574 scope.go:117] "RemoveContainer" containerID="59416758e499c8e902d89dad6840f360e8c65c02514dfe3af1e8d63b327df89f" Sep 08 04:17:58.798802 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:17:58.798784 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59416758e499c8e902d89dad6840f360e8c65c02514dfe3af1e8d63b327df89f\": container with ID starting with 59416758e499c8e902d89dad6840f360e8c65c02514dfe3af1e8d63b327df89f not found: ID does not exist" containerID="59416758e499c8e902d89dad6840f360e8c65c02514dfe3af1e8d63b327df89f" Sep 08 04:17:58.798866 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.798810 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59416758e499c8e902d89dad6840f360e8c65c02514dfe3af1e8d63b327df89f"} err="failed to get container status \"59416758e499c8e902d89dad6840f360e8c65c02514dfe3af1e8d63b327df89f\": rpc error: code = NotFound desc = could not find container \"59416758e499c8e902d89dad6840f360e8c65c02514dfe3af1e8d63b327df89f\": container with ID starting with 59416758e499c8e902d89dad6840f360e8c65c02514dfe3af1e8d63b327df89f not found: ID does not exist" Sep 08 04:17:58.799087 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.799068 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts"] Sep 08 04:17:58.803773 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.803752 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cb3759-5c88648594-mjwts"] Sep 08 04:17:58.815210 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.815186 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq"] Sep 08 04:17:58.823874 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:17:58.823847 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-cb3759-59f8bd5b65-jphmq"] Sep 08 04:18:00.368651 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:00.368617 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:00.455958 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:00.455919 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:00.568761 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:00.568723 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f400055-8987-42d5-a021-0b48600e6c8b" path="/var/lib/kubelet/pods/1f400055-8987-42d5-a021-0b48600e6c8b/volumes" Sep 08 04:18:00.569222 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:00.569201 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="862c4066-bdde-4949-a1f3-c078d0ac9da0" path="/var/lib/kubelet/pods/862c4066-bdde-4949-a1f3-c078d0ac9da0/volumes" Sep 08 04:18:01.601051 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:01.601016 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:01.671314 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:01.670491 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:01.930887 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:01.930793 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:02.216977 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.216897 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:02.312044 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.312013 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:02.335715 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.335680 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j"] Sep 08 04:18:02.336058 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.336044 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f400055-8987-42d5-a021-0b48600e6c8b" containerName="payload-processing" Sep 08 04:18:02.336103 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.336059 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f400055-8987-42d5-a021-0b48600e6c8b" containerName="payload-processing" Sep 08 04:18:02.336103 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.336067 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bf04b0d-fb46-413b-8312-82bf76b93186" containerName="revoke-keys" Sep 08 04:18:02.336103 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.336074 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bf04b0d-fb46-413b-8312-82bf76b93186" containerName="revoke-keys" Sep 08 04:18:02.336103 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.336086 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="862c4066-bdde-4949-a1f3-c078d0ac9da0" containerName="payload-pre-processing" Sep 08 04:18:02.336103 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.336092 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="862c4066-bdde-4949-a1f3-c078d0ac9da0" containerName="payload-pre-processing" Sep 08 04:18:02.336376 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.336141 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f400055-8987-42d5-a021-0b48600e6c8b" containerName="payload-processing" Sep 08 04:18:02.336376 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.336149 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bf04b0d-fb46-413b-8312-82bf76b93186" containerName="revoke-keys" Sep 08 04:18:02.336376 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.336159 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="862c4066-bdde-4949-a1f3-c078d0ac9da0" containerName="payload-pre-processing" Sep 08 04:18:02.340969 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.340949 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j" Sep 08 04:18:02.344274 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.344253 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7c5e63de-dockercfg-gp2mc\"" Sep 08 04:18:02.344526 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.344497 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7c5e63de\"" Sep 08 04:18:02.351041 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.351018 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j"] Sep 08 04:18:02.360707 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.360685 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc"] Sep 08 04:18:02.364106 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.364080 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc" Sep 08 04:18:02.384411 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.384375 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc"] Sep 08 04:18:02.463477 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.463440 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/583e207e-351b-438a-b200-b5536c3d6fcb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j\" (UID: \"583e207e-351b-438a-b200-b5536c3d6fcb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j" Sep 08 04:18:02.463685 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.463528 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fbb39c6-6afc-478b-8db9-0b31818ee42e-plugins-config-volume\") pod \"payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc\" (UID: \"6fbb39c6-6afc-478b-8db9-0b31818ee42e\") " pod="openshift-ingress/payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc" Sep 08 04:18:02.463685 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.463573 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxslv\" (UniqueName: \"kubernetes.io/projected/583e207e-351b-438a-b200-b5536c3d6fcb-kube-api-access-gxslv\") pod \"payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j\" (UID: \"583e207e-351b-438a-b200-b5536c3d6fcb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j" Sep 08 04:18:02.463804 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.463686 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlqs8\" (UniqueName: \"kubernetes.io/projected/6fbb39c6-6afc-478b-8db9-0b31818ee42e-kube-api-access-zlqs8\") pod \"payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc\" (UID: \"6fbb39c6-6afc-478b-8db9-0b31818ee42e\") " pod="openshift-ingress/payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc" Sep 08 04:18:02.564248 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.564215 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlqs8\" (UniqueName: \"kubernetes.io/projected/6fbb39c6-6afc-478b-8db9-0b31818ee42e-kube-api-access-zlqs8\") pod \"payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc\" (UID: \"6fbb39c6-6afc-478b-8db9-0b31818ee42e\") " pod="openshift-ingress/payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc" Sep 08 04:18:02.564439 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.564267 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/583e207e-351b-438a-b200-b5536c3d6fcb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j\" (UID: \"583e207e-351b-438a-b200-b5536c3d6fcb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j" Sep 08 04:18:02.564439 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.564300 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fbb39c6-6afc-478b-8db9-0b31818ee42e-plugins-config-volume\") pod \"payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc\" (UID: \"6fbb39c6-6afc-478b-8db9-0b31818ee42e\") " pod="openshift-ingress/payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc" Sep 08 04:18:02.564439 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.564318 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxslv\" (UniqueName: \"kubernetes.io/projected/583e207e-351b-438a-b200-b5536c3d6fcb-kube-api-access-gxslv\") pod \"payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j\" (UID: \"583e207e-351b-438a-b200-b5536c3d6fcb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j" Sep 08 04:18:02.565061 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.565039 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/583e207e-351b-438a-b200-b5536c3d6fcb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j\" (UID: \"583e207e-351b-438a-b200-b5536c3d6fcb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j" Sep 08 04:18:02.565208 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.565187 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fbb39c6-6afc-478b-8db9-0b31818ee42e-plugins-config-volume\") pod \"payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc\" (UID: \"6fbb39c6-6afc-478b-8db9-0b31818ee42e\") " pod="openshift-ingress/payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc" Sep 08 04:18:02.587999 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.587964 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlqs8\" (UniqueName: \"kubernetes.io/projected/6fbb39c6-6afc-478b-8db9-0b31818ee42e-kube-api-access-zlqs8\") pod \"payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc\" (UID: \"6fbb39c6-6afc-478b-8db9-0b31818ee42e\") " pod="openshift-ingress/payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc" Sep 08 04:18:02.588129 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.588064 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxslv\" (UniqueName: \"kubernetes.io/projected/583e207e-351b-438a-b200-b5536c3d6fcb-kube-api-access-gxslv\") pod \"payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j\" (UID: \"583e207e-351b-438a-b200-b5536c3d6fcb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j" Sep 08 04:18:02.652410 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.652367 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j" Sep 08 04:18:02.673418 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.673390 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc" Sep 08 04:18:02.927416 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.927388 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j"] Sep 08 04:18:02.928600 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:18:02.928574 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583e207e_351b_438a_b200_b5536c3d6fcb.slice/crio-bda91f95e80b74a0160e78875b2375bb263d233b6290b1985d54ea036116490e WatchSource:0}: Error finding container bda91f95e80b74a0160e78875b2375bb263d233b6290b1985d54ea036116490e: Status 404 returned error can't find the container with id bda91f95e80b74a0160e78875b2375bb263d233b6290b1985d54ea036116490e Sep 08 04:18:02.971583 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:02.971544 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc"] Sep 08 04:18:02.971704 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:18:02.971574 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fbb39c6_6afc_478b_8db9_0b31818ee42e.slice/crio-d4af6089b75b033b71f12d1287c6571266d14a7daa9a6f1c31f87a423789496a WatchSource:0}: Error finding container d4af6089b75b033b71f12d1287c6571266d14a7daa9a6f1c31f87a423789496a: Status 404 returned error can't find the container with id d4af6089b75b033b71f12d1287c6571266d14a7daa9a6f1c31f87a423789496a Sep 08 04:18:03.813125 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:03.813070 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j" event={"ID":"583e207e-351b-438a-b200-b5536c3d6fcb","Type":"ContainerStarted","Data":"575c4394a3441bbc58db20c7ebda479d645f09456c86168ea6c759d6c993e72c"} Sep 08 04:18:03.813624 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:03.813138 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j" event={"ID":"583e207e-351b-438a-b200-b5536c3d6fcb","Type":"ContainerStarted","Data":"bda91f95e80b74a0160e78875b2375bb263d233b6290b1985d54ea036116490e"} Sep 08 04:18:03.813624 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:03.813167 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j" Sep 08 04:18:03.814696 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:03.814665 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc" event={"ID":"6fbb39c6-6afc-478b-8db9-0b31818ee42e","Type":"ContainerStarted","Data":"d35742ae7e27d4d7d65b050342ad96a691b2c0f49b2088bbc2ec03986634c4c0"} Sep 08 04:18:03.814820 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:03.814757 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc" Sep 08 04:18:03.814820 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:03.814786 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc" event={"ID":"6fbb39c6-6afc-478b-8db9-0b31818ee42e","Type":"ContainerStarted","Data":"d4af6089b75b033b71f12d1287c6571266d14a7daa9a6f1c31f87a423789496a"} Sep 08 04:18:03.828843 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:03.828798 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j" podStartSLOduration=1.828784236 podStartE2EDuration="1.828784236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:18:03.827588744 +0000 UTC m=+1057.750049507" watchObservedRunningTime="2026-09-08 04:18:03.828784236 +0000 UTC m=+1057.751244976" Sep 08 04:18:03.845273 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:03.845209 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc" podStartSLOduration=1.8451925930000002 podStartE2EDuration="1.845192593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:18:03.843029498 +0000 UTC m=+1057.765490259" watchObservedRunningTime="2026-09-08 04:18:03.845192593 +0000 UTC m=+1057.767653372" Sep 08 04:18:04.928559 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:04.928526 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:05.024370 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:05.024330 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:05.155492 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:05.155451 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962"] Sep 08 04:18:05.155705 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:05.155670 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" podUID="00d03860-2980-434f-a45b-d41427acc87c" containerName="istio-proxy" containerID="cri-o://c8fa18c6bb66d9033672de931f4cd4d038258d3b5368356ec7ceb4ccafacae30" gracePeriod=30 Sep 08 04:18:05.177522 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:05.177482 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:05.614703 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:05.614665 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:05.951447 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:05.951363 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:06.050242 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:06.050196 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:07.757153 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:07.757108 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:07.846834 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:07.846805 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:07.949946 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:07.949912 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:10.414803 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.414777 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:18:10.543946 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.543909 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-979dx\" (UniqueName: \"kubernetes.io/projected/00d03860-2980-434f-a45b-d41427acc87c-kube-api-access-979dx\") pod \"00d03860-2980-434f-a45b-d41427acc87c\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " Sep 08 04:18:10.543946 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.543953 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-workload-certs\") pod \"00d03860-2980-434f-a45b-d41427acc87c\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " Sep 08 04:18:10.544185 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.543975 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-workload-socket\") pod \"00d03860-2980-434f-a45b-d41427acc87c\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " Sep 08 04:18:10.544185 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.544010 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-istio-data\") pod \"00d03860-2980-434f-a45b-d41427acc87c\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " Sep 08 04:18:10.544185 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.544033 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-istio-envoy\") pod \"00d03860-2980-434f-a45b-d41427acc87c\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " Sep 08 04:18:10.544185 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.544047 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-credential-socket\") pod \"00d03860-2980-434f-a45b-d41427acc87c\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " Sep 08 04:18:10.544185 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.544063 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d03860-2980-434f-a45b-d41427acc87c-istiod-ca-cert\") pod \"00d03860-2980-434f-a45b-d41427acc87c\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " Sep 08 04:18:10.544185 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.544085 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d03860-2980-434f-a45b-d41427acc87c-istio-podinfo\") pod \"00d03860-2980-434f-a45b-d41427acc87c\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " Sep 08 04:18:10.544458 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.544184 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d03860-2980-434f-a45b-d41427acc87c-istio-token\") pod \"00d03860-2980-434f-a45b-d41427acc87c\" (UID: \"00d03860-2980-434f-a45b-d41427acc87c\") " Sep 08 04:18:10.544458 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.544211 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "00d03860-2980-434f-a45b-d41427acc87c" (UID: "00d03860-2980-434f-a45b-d41427acc87c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:18:10.544458 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.544378 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-istio-data" (OuterVolumeSpecName: "istio-data") pod "00d03860-2980-434f-a45b-d41427acc87c" (UID: "00d03860-2980-434f-a45b-d41427acc87c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:18:10.544458 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.544395 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "00d03860-2980-434f-a45b-d41427acc87c" (UID: "00d03860-2980-434f-a45b-d41427acc87c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:18:10.544458 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.544407 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "00d03860-2980-434f-a45b-d41427acc87c" (UID: "00d03860-2980-434f-a45b-d41427acc87c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:18:10.544983 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.544499 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-workload-certs\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:18:10.544983 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.544549 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-workload-socket\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:18:10.544983 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.544558 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-istio-data\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:18:10.544983 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.544567 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-credential-socket\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:18:10.544983 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.544694 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00d03860-2980-434f-a45b-d41427acc87c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "00d03860-2980-434f-a45b-d41427acc87c" (UID: "00d03860-2980-434f-a45b-d41427acc87c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:18:10.546428 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.546397 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00d03860-2980-434f-a45b-d41427acc87c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "00d03860-2980-434f-a45b-d41427acc87c" (UID: "00d03860-2980-434f-a45b-d41427acc87c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 04:18:10.546428 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.546411 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d03860-2980-434f-a45b-d41427acc87c-kube-api-access-979dx" (OuterVolumeSpecName: "kube-api-access-979dx") pod "00d03860-2980-434f-a45b-d41427acc87c" (UID: "00d03860-2980-434f-a45b-d41427acc87c"). InnerVolumeSpecName "kube-api-access-979dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:18:10.546597 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.546489 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d03860-2980-434f-a45b-d41427acc87c-istio-token" (OuterVolumeSpecName: "istio-token") pod "00d03860-2980-434f-a45b-d41427acc87c" (UID: "00d03860-2980-434f-a45b-d41427acc87c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:18:10.546597 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.546525 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "00d03860-2980-434f-a45b-d41427acc87c" (UID: "00d03860-2980-434f-a45b-d41427acc87c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:18:10.645886 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.645850 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00d03860-2980-434f-a45b-d41427acc87c-istio-envoy\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:18:10.645886 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.645885 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00d03860-2980-434f-a45b-d41427acc87c-istiod-ca-cert\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:18:10.646140 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.645898 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00d03860-2980-434f-a45b-d41427acc87c-istio-podinfo\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:18:10.646140 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.645912 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00d03860-2980-434f-a45b-d41427acc87c-istio-token\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:18:10.646140 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.645924 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-979dx\" (UniqueName: \"kubernetes.io/projected/00d03860-2980-434f-a45b-d41427acc87c-kube-api-access-979dx\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:18:10.857684 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.857589 2574 generic.go:356] "Generic (PLEG): container finished" podID="00d03860-2980-434f-a45b-d41427acc87c" containerID="c8fa18c6bb66d9033672de931f4cd4d038258d3b5368356ec7ceb4ccafacae30" exitCode=0 Sep 08 04:18:10.857684 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.857639 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" Sep 08 04:18:10.857684 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.857667 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" event={"ID":"00d03860-2980-434f-a45b-d41427acc87c","Type":"ContainerDied","Data":"c8fa18c6bb66d9033672de931f4cd4d038258d3b5368356ec7ceb4ccafacae30"} Sep 08 04:18:10.857935 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.857716 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962" event={"ID":"00d03860-2980-434f-a45b-d41427acc87c","Type":"ContainerDied","Data":"f47aed1d1c8b702d5f7b67fea2660e2f7b52998bd0ed3652c12658e4d160d1e1"} Sep 08 04:18:10.857935 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.857742 2574 scope.go:117] "RemoveContainer" containerID="c8fa18c6bb66d9033672de931f4cd4d038258d3b5368356ec7ceb4ccafacae30" Sep 08 04:18:10.866719 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.866700 2574 scope.go:117] "RemoveContainer" containerID="c992d29a8708f537979591350b749c37b875038a3640d9bb00c8e257864ac738" Sep 08 04:18:10.878562 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.878540 2574 scope.go:117] "RemoveContainer" containerID="c8fa18c6bb66d9033672de931f4cd4d038258d3b5368356ec7ceb4ccafacae30" Sep 08 04:18:10.878978 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:18:10.878956 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8fa18c6bb66d9033672de931f4cd4d038258d3b5368356ec7ceb4ccafacae30\": container with ID starting with c8fa18c6bb66d9033672de931f4cd4d038258d3b5368356ec7ceb4ccafacae30 not found: ID does not exist" containerID="c8fa18c6bb66d9033672de931f4cd4d038258d3b5368356ec7ceb4ccafacae30" Sep 08 04:18:10.879040 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.878987 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8fa18c6bb66d9033672de931f4cd4d038258d3b5368356ec7ceb4ccafacae30"} err="failed to get container status \"c8fa18c6bb66d9033672de931f4cd4d038258d3b5368356ec7ceb4ccafacae30\": rpc error: code = NotFound desc = could not find container \"c8fa18c6bb66d9033672de931f4cd4d038258d3b5368356ec7ceb4ccafacae30\": container with ID starting with c8fa18c6bb66d9033672de931f4cd4d038258d3b5368356ec7ceb4ccafacae30 not found: ID does not exist" Sep 08 04:18:10.879040 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.879005 2574 scope.go:117] "RemoveContainer" containerID="c992d29a8708f537979591350b749c37b875038a3640d9bb00c8e257864ac738" Sep 08 04:18:10.879302 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:18:10.879285 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c992d29a8708f537979591350b749c37b875038a3640d9bb00c8e257864ac738\": container with ID starting with c992d29a8708f537979591350b749c37b875038a3640d9bb00c8e257864ac738 not found: ID does not exist" containerID="c992d29a8708f537979591350b749c37b875038a3640d9bb00c8e257864ac738" Sep 08 04:18:10.879343 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.879312 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962"] Sep 08 04:18:10.879343 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.879309 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c992d29a8708f537979591350b749c37b875038a3640d9bb00c8e257864ac738"} err="failed to get container status \"c992d29a8708f537979591350b749c37b875038a3640d9bb00c8e257864ac738\": rpc error: code = NotFound desc = could not find container \"c992d29a8708f537979591350b749c37b875038a3640d9bb00c8e257864ac738\": container with ID starting with c992d29a8708f537979591350b749c37b875038a3640d9bb00c8e257864ac738 not found: ID does not exist" Sep 08 04:18:10.883245 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:10.883225 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-cb3759-openshift-default-5b6f7b5976-tn962"] Sep 08 04:18:12.568662 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:12.568626 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00d03860-2980-434f-a45b-d41427acc87c" path="/var/lib/kubelet/pods/00d03860-2980-434f-a45b-d41427acc87c/volumes" Sep 08 04:18:14.821692 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:14.821664 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc" Sep 08 04:18:14.822229 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:14.821787 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j" Sep 08 04:18:22.306140 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:22.306062 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:25.165491 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:25.165435 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:25.294408 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:25.294376 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:25.396368 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:25.396335 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:25.500277 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:25.500248 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:25.597261 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:25.597222 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:26.016315 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:26.016287 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:26.113321 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:26.113290 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:26.431310 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:26.431232 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:30.669140 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:30.669097 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:34.295357 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:34.295325 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:34.459148 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:34.459118 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:34.643200 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:34.643136 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c5e63de-kwjfw"] Sep 08 04:18:34.643518 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:34.643487 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00d03860-2980-434f-a45b-d41427acc87c" containerName="istio-proxy" Sep 08 04:18:34.643518 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:34.643516 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="00d03860-2980-434f-a45b-d41427acc87c" containerName="istio-proxy" Sep 08 04:18:34.643603 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:34.643530 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00d03860-2980-434f-a45b-d41427acc87c" containerName="istio-proxy" Sep 08 04:18:34.643603 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:34.643536 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="00d03860-2980-434f-a45b-d41427acc87c" containerName="istio-proxy" Sep 08 04:18:34.643603 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:34.643595 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="00d03860-2980-434f-a45b-d41427acc87c" containerName="istio-proxy" Sep 08 04:18:34.643603 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:34.643603 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="00d03860-2980-434f-a45b-d41427acc87c" containerName="istio-proxy" Sep 08 04:18:34.648016 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:34.647999 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c5e63de-kwjfw" Sep 08 04:18:34.651311 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:34.651293 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7r29w\"" Sep 08 04:18:34.651575 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:34.651562 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 04:18:34.664008 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:34.663982 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddcae6e-8f9d-4526-92a9-7d8feab4d0f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7c5e63de-kwjfw\" (UID: \"7ddcae6e-8f9d-4526-92a9-7d8feab4d0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c5e63de-kwjfw" Sep 08 04:18:34.664170 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:34.664151 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c5e63de-kwjfw"] Sep 08 04:18:34.765111 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:34.765082 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddcae6e-8f9d-4526-92a9-7d8feab4d0f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7c5e63de-kwjfw\" (UID: \"7ddcae6e-8f9d-4526-92a9-7d8feab4d0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c5e63de-kwjfw" Sep 08 04:18:34.765697 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:34.765675 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddcae6e-8f9d-4526-92a9-7d8feab4d0f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7c5e63de-kwjfw\" (UID: \"7ddcae6e-8f9d-4526-92a9-7d8feab4d0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c5e63de-kwjfw" Sep 08 04:18:34.957962 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:34.957903 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c5e63de-kwjfw" Sep 08 04:18:35.249083 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:35.249060 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c5e63de-kwjfw"] Sep 08 04:18:35.250944 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:18:35.250917 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ddcae6e_8f9d_4526_92a9_7d8feab4d0f4.slice/crio-ff6556f8d392e65814a080b06c4bfcb1b438fd34b9487967c6ddb8b41b021832 WatchSource:0}: Error finding container ff6556f8d392e65814a080b06c4bfcb1b438fd34b9487967c6ddb8b41b021832: Status 404 returned error can't find the container with id ff6556f8d392e65814a080b06c4bfcb1b438fd34b9487967c6ddb8b41b021832 Sep 08 04:18:36.003136 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:36.003105 2574 generic.go:356] "Generic (PLEG): container finished" podID="7ddcae6e-8f9d-4526-92a9-7d8feab4d0f4" containerID="ca551dc1061c8965158e55752054f732b2826688183fd61bb4a33e60978b0b47" exitCode=0 Sep 08 04:18:36.003443 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:36.003164 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c5e63de-kwjfw" event={"ID":"7ddcae6e-8f9d-4526-92a9-7d8feab4d0f4","Type":"ContainerDied","Data":"ca551dc1061c8965158e55752054f732b2826688183fd61bb4a33e60978b0b47"} Sep 08 04:18:36.003443 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:36.003201 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c5e63de-kwjfw" event={"ID":"7ddcae6e-8f9d-4526-92a9-7d8feab4d0f4","Type":"ContainerStarted","Data":"ff6556f8d392e65814a080b06c4bfcb1b438fd34b9487967c6ddb8b41b021832"} Sep 08 04:18:37.139976 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:37.139954 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c5e63de-kwjfw" Sep 08 04:18:37.183085 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:37.183060 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddcae6e-8f9d-4526-92a9-7d8feab4d0f4-maas-api-service-ca\") pod \"7ddcae6e-8f9d-4526-92a9-7d8feab4d0f4\" (UID: \"7ddcae6e-8f9d-4526-92a9-7d8feab4d0f4\") " Sep 08 04:18:37.183400 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:37.183381 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ddcae6e-8f9d-4526-92a9-7d8feab4d0f4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7ddcae6e-8f9d-4526-92a9-7d8feab4d0f4" (UID: "7ddcae6e-8f9d-4526-92a9-7d8feab4d0f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:18:37.283692 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:37.283640 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddcae6e-8f9d-4526-92a9-7d8feab4d0f4-maas-api-service-ca\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:18:38.016074 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:38.016045 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c5e63de-kwjfw" Sep 08 04:18:38.016217 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:38.016088 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c5e63de-kwjfw" event={"ID":"7ddcae6e-8f9d-4526-92a9-7d8feab4d0f4","Type":"ContainerDied","Data":"ff6556f8d392e65814a080b06c4bfcb1b438fd34b9487967c6ddb8b41b021832"} Sep 08 04:18:38.016217 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:38.016135 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff6556f8d392e65814a080b06c4bfcb1b438fd34b9487967c6ddb8b41b021832" Sep 08 04:18:38.021047 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:38.021011 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:38.951413 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:38.951367 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:39.068255 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:39.068220 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0f3e-fz6dp"] Sep 08 04:18:39.068700 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:39.068683 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ddcae6e-8f9d-4526-92a9-7d8feab4d0f4" containerName="revoke-keys" Sep 08 04:18:39.068700 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:39.068700 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ddcae6e-8f9d-4526-92a9-7d8feab4d0f4" containerName="revoke-keys" Sep 08 04:18:39.068858 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:39.068760 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ddcae6e-8f9d-4526-92a9-7d8feab4d0f4" containerName="revoke-keys" Sep 08 04:18:39.071872 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:39.071850 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0f3e-fz6dp" Sep 08 04:18:39.074920 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:39.074898 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7r29w\"" Sep 08 04:18:39.075019 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:39.074937 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 04:18:39.095025 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:39.095002 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0f3e-fz6dp"] Sep 08 04:18:39.098401 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:39.098378 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3791a8ca-9381-49f8-96a0-314431703eb7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ff0f3e-fz6dp\" (UID: \"3791a8ca-9381-49f8-96a0-314431703eb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0f3e-fz6dp" Sep 08 04:18:39.199460 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:39.199433 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3791a8ca-9381-49f8-96a0-314431703eb7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ff0f3e-fz6dp\" (UID: \"3791a8ca-9381-49f8-96a0-314431703eb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0f3e-fz6dp" Sep 08 04:18:39.200034 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:39.200016 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3791a8ca-9381-49f8-96a0-314431703eb7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ff0f3e-fz6dp\" (UID: \"3791a8ca-9381-49f8-96a0-314431703eb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0f3e-fz6dp" Sep 08 04:18:39.267213 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:39.267186 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:39.382203 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:39.382176 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0f3e-fz6dp" Sep 08 04:18:39.661863 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:39.661837 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0f3e-fz6dp"] Sep 08 04:18:39.662047 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:18:39.662028 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3791a8ca_9381_49f8_96a0_314431703eb7.slice/crio-8ad8c2c1aeabcd2d0b3598b2dfbbfb36abbe2c41159fba5cf75c22f95fcdf4de WatchSource:0}: Error finding container 8ad8c2c1aeabcd2d0b3598b2dfbbfb36abbe2c41159fba5cf75c22f95fcdf4de: Status 404 returned error can't find the container with id 8ad8c2c1aeabcd2d0b3598b2dfbbfb36abbe2c41159fba5cf75c22f95fcdf4de Sep 08 04:18:40.032483 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:40.032453 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0f3e-fz6dp" event={"ID":"3791a8ca-9381-49f8-96a0-314431703eb7","Type":"ContainerStarted","Data":"55071f61f5b9e527f79f3605bbc6d38f462416a6ad93cc592f5ce0f8a34a6c67"} Sep 08 04:18:40.032797 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:40.032553 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0f3e-fz6dp" event={"ID":"3791a8ca-9381-49f8-96a0-314431703eb7","Type":"ContainerStarted","Data":"8ad8c2c1aeabcd2d0b3598b2dfbbfb36abbe2c41159fba5cf75c22f95fcdf4de"} Sep 08 04:18:40.056982 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:40.056939 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0f3e-fz6dp" podStartSLOduration=1.056925446 podStartE2EDuration="1.056925446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:18:40.052855574 +0000 UTC m=+1093.975316333" watchObservedRunningTime="2026-09-08 04:18:40.056925446 +0000 UTC m=+1093.979386186" Sep 08 04:18:41.039350 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:41.039314 2574 generic.go:356] "Generic (PLEG): container finished" podID="3791a8ca-9381-49f8-96a0-314431703eb7" containerID="55071f61f5b9e527f79f3605bbc6d38f462416a6ad93cc592f5ce0f8a34a6c67" exitCode=0 Sep 08 04:18:41.039782 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:41.039397 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0f3e-fz6dp" event={"ID":"3791a8ca-9381-49f8-96a0-314431703eb7","Type":"ContainerDied","Data":"55071f61f5b9e527f79f3605bbc6d38f462416a6ad93cc592f5ce0f8a34a6c67"} Sep 08 04:18:42.176329 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:42.176307 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0f3e-fz6dp" Sep 08 04:18:42.220129 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:42.220103 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3791a8ca-9381-49f8-96a0-314431703eb7-maas-api-service-ca\") pod \"3791a8ca-9381-49f8-96a0-314431703eb7\" (UID: \"3791a8ca-9381-49f8-96a0-314431703eb7\") " Sep 08 04:18:42.220437 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:42.220416 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3791a8ca-9381-49f8-96a0-314431703eb7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3791a8ca-9381-49f8-96a0-314431703eb7" (UID: "3791a8ca-9381-49f8-96a0-314431703eb7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:18:42.321662 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:42.321610 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3791a8ca-9381-49f8-96a0-314431703eb7-maas-api-service-ca\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:18:43.055446 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:43.055415 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0f3e-fz6dp" Sep 08 04:18:43.055635 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:43.055459 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0f3e-fz6dp" event={"ID":"3791a8ca-9381-49f8-96a0-314431703eb7","Type":"ContainerDied","Data":"8ad8c2c1aeabcd2d0b3598b2dfbbfb36abbe2c41159fba5cf75c22f95fcdf4de"} Sep 08 04:18:43.055635 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:43.055541 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ad8c2c1aeabcd2d0b3598b2dfbbfb36abbe2c41159fba5cf75c22f95fcdf4de" Sep 08 04:18:44.797028 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:44.796987 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:45.583718 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:45.583688 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:45.620333 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:45.620301 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc"] Sep 08 04:18:45.620601 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:45.620572 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc" podUID="6fbb39c6-6afc-478b-8db9-0b31818ee42e" containerName="payload-processing" containerID="cri-o://d35742ae7e27d4d7d65b050342ad96a691b2c0f49b2088bbc2ec03986634c4c0" gracePeriod=30 Sep 08 04:18:45.627852 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:45.627828 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j"] Sep 08 04:18:45.628434 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:45.628407 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j" podUID="583e207e-351b-438a-b200-b5536c3d6fcb" containerName="payload-pre-processing" containerID="cri-o://575c4394a3441bbc58db20c7ebda479d645f09456c86168ea6c759d6c993e72c" gracePeriod=30 Sep 08 04:18:45.683118 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:45.683084 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:45.770924 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:45.770889 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:45.887695 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:45.887668 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:45.905323 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:45.905294 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c5e63de-kwjfw"] Sep 08 04:18:45.905479 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:45.905357 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c5e63de-kwjfw"] Sep 08 04:18:45.921438 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:45.921419 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc" Sep 08 04:18:45.924393 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:45.924375 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j" Sep 08 04:18:45.951687 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:45.951653 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlqs8\" (UniqueName: \"kubernetes.io/projected/6fbb39c6-6afc-478b-8db9-0b31818ee42e-kube-api-access-zlqs8\") pod \"6fbb39c6-6afc-478b-8db9-0b31818ee42e\" (UID: \"6fbb39c6-6afc-478b-8db9-0b31818ee42e\") " Sep 08 04:18:45.952002 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:45.951979 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/583e207e-351b-438a-b200-b5536c3d6fcb-plugins-config-volume\") pod \"583e207e-351b-438a-b200-b5536c3d6fcb\" (UID: \"583e207e-351b-438a-b200-b5536c3d6fcb\") " Sep 08 04:18:45.952155 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:45.952139 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fbb39c6-6afc-478b-8db9-0b31818ee42e-plugins-config-volume\") pod \"6fbb39c6-6afc-478b-8db9-0b31818ee42e\" (UID: \"6fbb39c6-6afc-478b-8db9-0b31818ee42e\") " Sep 08 04:18:45.952284 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:45.952267 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxslv\" (UniqueName: \"kubernetes.io/projected/583e207e-351b-438a-b200-b5536c3d6fcb-kube-api-access-gxslv\") pod \"583e207e-351b-438a-b200-b5536c3d6fcb\" (UID: \"583e207e-351b-438a-b200-b5536c3d6fcb\") " Sep 08 04:18:45.952835 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:45.952807 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583e207e-351b-438a-b200-b5536c3d6fcb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "583e207e-351b-438a-b200-b5536c3d6fcb" (UID: "583e207e-351b-438a-b200-b5536c3d6fcb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:18:45.952971 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:45.952949 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fbb39c6-6afc-478b-8db9-0b31818ee42e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6fbb39c6-6afc-478b-8db9-0b31818ee42e" (UID: "6fbb39c6-6afc-478b-8db9-0b31818ee42e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:18:45.954985 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:45.954962 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fbb39c6-6afc-478b-8db9-0b31818ee42e-kube-api-access-zlqs8" (OuterVolumeSpecName: "kube-api-access-zlqs8") pod "6fbb39c6-6afc-478b-8db9-0b31818ee42e" (UID: "6fbb39c6-6afc-478b-8db9-0b31818ee42e"). InnerVolumeSpecName "kube-api-access-zlqs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:18:45.955129 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:45.955100 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583e207e-351b-438a-b200-b5536c3d6fcb-kube-api-access-gxslv" (OuterVolumeSpecName: "kube-api-access-gxslv") pod "583e207e-351b-438a-b200-b5536c3d6fcb" (UID: "583e207e-351b-438a-b200-b5536c3d6fcb"). InnerVolumeSpecName "kube-api-access-gxslv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:18:46.012810 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:46.012782 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:46.054061 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:46.054036 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/583e207e-351b-438a-b200-b5536c3d6fcb-plugins-config-volume\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:18:46.054186 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:46.054062 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fbb39c6-6afc-478b-8db9-0b31818ee42e-plugins-config-volume\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:18:46.054186 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:46.054078 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxslv\" (UniqueName: \"kubernetes.io/projected/583e207e-351b-438a-b200-b5536c3d6fcb-kube-api-access-gxslv\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:18:46.054186 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:46.054091 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlqs8\" (UniqueName: \"kubernetes.io/projected/6fbb39c6-6afc-478b-8db9-0b31818ee42e-kube-api-access-zlqs8\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:18:46.075093 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:46.075063 2574 generic.go:356] "Generic (PLEG): container finished" podID="583e207e-351b-438a-b200-b5536c3d6fcb" containerID="575c4394a3441bbc58db20c7ebda479d645f09456c86168ea6c759d6c993e72c" exitCode=0 Sep 08 04:18:46.075197 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:46.075103 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j" Sep 08 04:18:46.075197 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:46.075148 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j" event={"ID":"583e207e-351b-438a-b200-b5536c3d6fcb","Type":"ContainerDied","Data":"575c4394a3441bbc58db20c7ebda479d645f09456c86168ea6c759d6c993e72c"} Sep 08 04:18:46.075290 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:46.075244 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j" event={"ID":"583e207e-351b-438a-b200-b5536c3d6fcb","Type":"ContainerDied","Data":"bda91f95e80b74a0160e78875b2375bb263d233b6290b1985d54ea036116490e"} Sep 08 04:18:46.075290 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:46.075271 2574 scope.go:117] "RemoveContainer" containerID="575c4394a3441bbc58db20c7ebda479d645f09456c86168ea6c759d6c993e72c" Sep 08 04:18:46.076715 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:46.076691 2574 generic.go:356] "Generic (PLEG): container finished" podID="6fbb39c6-6afc-478b-8db9-0b31818ee42e" containerID="d35742ae7e27d4d7d65b050342ad96a691b2c0f49b2088bbc2ec03986634c4c0" exitCode=0 Sep 08 04:18:46.076833 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:46.076725 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc" Sep 08 04:18:46.076833 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:46.076803 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc" event={"ID":"6fbb39c6-6afc-478b-8db9-0b31818ee42e","Type":"ContainerDied","Data":"d35742ae7e27d4d7d65b050342ad96a691b2c0f49b2088bbc2ec03986634c4c0"} Sep 08 04:18:46.076938 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:46.076855 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc" event={"ID":"6fbb39c6-6afc-478b-8db9-0b31818ee42e","Type":"ContainerDied","Data":"d4af6089b75b033b71f12d1287c6571266d14a7daa9a6f1c31f87a423789496a"} Sep 08 04:18:46.084619 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:46.084602 2574 scope.go:117] "RemoveContainer" containerID="575c4394a3441bbc58db20c7ebda479d645f09456c86168ea6c759d6c993e72c" Sep 08 04:18:46.084944 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:18:46.084923 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"575c4394a3441bbc58db20c7ebda479d645f09456c86168ea6c759d6c993e72c\": container with ID starting with 575c4394a3441bbc58db20c7ebda479d645f09456c86168ea6c759d6c993e72c not found: ID does not exist" containerID="575c4394a3441bbc58db20c7ebda479d645f09456c86168ea6c759d6c993e72c" Sep 08 04:18:46.085024 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:46.084949 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"575c4394a3441bbc58db20c7ebda479d645f09456c86168ea6c759d6c993e72c"} err="failed to get container status \"575c4394a3441bbc58db20c7ebda479d645f09456c86168ea6c759d6c993e72c\": rpc error: code = NotFound desc = could not find container \"575c4394a3441bbc58db20c7ebda479d645f09456c86168ea6c759d6c993e72c\": container with ID starting with 575c4394a3441bbc58db20c7ebda479d645f09456c86168ea6c759d6c993e72c not found: ID does not exist" Sep 08 04:18:46.085024 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:46.084967 2574 scope.go:117] "RemoveContainer" containerID="d35742ae7e27d4d7d65b050342ad96a691b2c0f49b2088bbc2ec03986634c4c0" Sep 08 04:18:46.094262 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:46.094245 2574 scope.go:117] "RemoveContainer" containerID="d35742ae7e27d4d7d65b050342ad96a691b2c0f49b2088bbc2ec03986634c4c0" Sep 08 04:18:46.094559 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:18:46.094542 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d35742ae7e27d4d7d65b050342ad96a691b2c0f49b2088bbc2ec03986634c4c0\": container with ID starting with d35742ae7e27d4d7d65b050342ad96a691b2c0f49b2088bbc2ec03986634c4c0 not found: ID does not exist" containerID="d35742ae7e27d4d7d65b050342ad96a691b2c0f49b2088bbc2ec03986634c4c0" Sep 08 04:18:46.094622 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:46.094565 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d35742ae7e27d4d7d65b050342ad96a691b2c0f49b2088bbc2ec03986634c4c0"} err="failed to get container status \"d35742ae7e27d4d7d65b050342ad96a691b2c0f49b2088bbc2ec03986634c4c0\": rpc error: code = NotFound desc = could not find container \"d35742ae7e27d4d7d65b050342ad96a691b2c0f49b2088bbc2ec03986634c4c0\": container with ID starting with d35742ae7e27d4d7d65b050342ad96a691b2c0f49b2088bbc2ec03986634c4c0 not found: ID does not exist" Sep 08 04:18:46.115295 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:46.115277 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc"] Sep 08 04:18:46.119166 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:46.119142 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7c5e63de-656f8b5665-wgmdc"] Sep 08 04:18:46.141547 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:46.141483 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j"] Sep 08 04:18:46.146835 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:46.146813 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7c5e63de-d4769df7d-98g7j"] Sep 08 04:18:46.375639 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:46.375613 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:46.567663 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:46.567637 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="583e207e-351b-438a-b200-b5536c3d6fcb" path="/var/lib/kubelet/pods/583e207e-351b-438a-b200-b5536c3d6fcb/volumes" Sep 08 04:18:46.568056 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:46.568037 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fbb39c6-6afc-478b-8db9-0b31818ee42e" path="/var/lib/kubelet/pods/6fbb39c6-6afc-478b-8db9-0b31818ee42e/volumes" Sep 08 04:18:46.568406 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:46.568387 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ddcae6e-8f9d-4526-92a9-7d8feab4d0f4" path="/var/lib/kubelet/pods/7ddcae6e-8f9d-4526-92a9-7d8feab4d0f4/volumes" Sep 08 04:18:49.505881 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:49.505838 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:50.534337 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:50.534302 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ff0f3e-29814015-qqvnl"] Sep 08 04:18:50.540361 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:50.540331 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ff0f3e-29814015-qqvnl"] Sep 08 04:18:50.545633 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:50.545605 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:50.567178 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:50.567151 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c40f5758-975c-47cc-9e91-095556297aa6" path="/var/lib/kubelet/pods/c40f5758-975c-47cc-9e91-095556297aa6/volumes" Sep 08 04:18:50.592069 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:50.592043 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7"] Sep 08 04:18:50.592337 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:50.592310 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7" podUID="fc7eeac0-8c21-49d3-b9f1-43f850df2cea" containerName="payload-processing" containerID="cri-o://e74babdef86ee92638a9c996d22609a7bed987f77ef671f2ea0d1e9d9bbae91c" gracePeriod=30 Sep 08 04:18:50.614652 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:50.614628 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c"] Sep 08 04:18:50.614820 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:50.614802 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c" podUID="b6973da3-09ec-4aa2-a0dd-e525a8289ca8" containerName="payload-pre-processing" containerID="cri-o://90e3f0910ea20d2df22fa5da19b54c144c1c8906f17ec3e0d775a31646291598" gracePeriod=30 Sep 08 04:18:50.757603 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:50.757548 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:50.885236 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:50.885201 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7" Sep 08 04:18:50.889882 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:50.889846 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:50.905974 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:50.905952 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c" Sep 08 04:18:50.995995 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:50.995966 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qsj9\" (UniqueName: \"kubernetes.io/projected/b6973da3-09ec-4aa2-a0dd-e525a8289ca8-kube-api-access-9qsj9\") pod \"b6973da3-09ec-4aa2-a0dd-e525a8289ca8\" (UID: \"b6973da3-09ec-4aa2-a0dd-e525a8289ca8\") " Sep 08 04:18:50.996189 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:50.996102 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zl8h\" (UniqueName: \"kubernetes.io/projected/fc7eeac0-8c21-49d3-b9f1-43f850df2cea-kube-api-access-2zl8h\") pod \"fc7eeac0-8c21-49d3-b9f1-43f850df2cea\" (UID: \"fc7eeac0-8c21-49d3-b9f1-43f850df2cea\") " Sep 08 04:18:50.996189 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:50.996151 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7eeac0-8c21-49d3-b9f1-43f850df2cea-plugins-config-volume\") pod \"fc7eeac0-8c21-49d3-b9f1-43f850df2cea\" (UID: \"fc7eeac0-8c21-49d3-b9f1-43f850df2cea\") " Sep 08 04:18:50.996307 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:50.996198 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6973da3-09ec-4aa2-a0dd-e525a8289ca8-plugins-config-volume\") pod \"b6973da3-09ec-4aa2-a0dd-e525a8289ca8\" (UID: \"b6973da3-09ec-4aa2-a0dd-e525a8289ca8\") " Sep 08 04:18:50.996621 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:50.996590 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6973da3-09ec-4aa2-a0dd-e525a8289ca8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b6973da3-09ec-4aa2-a0dd-e525a8289ca8" (UID: "b6973da3-09ec-4aa2-a0dd-e525a8289ca8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:18:50.996749 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:50.996617 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc7eeac0-8c21-49d3-b9f1-43f850df2cea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fc7eeac0-8c21-49d3-b9f1-43f850df2cea" (UID: "fc7eeac0-8c21-49d3-b9f1-43f850df2cea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:18:50.998158 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:50.998137 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc7eeac0-8c21-49d3-b9f1-43f850df2cea-kube-api-access-2zl8h" (OuterVolumeSpecName: "kube-api-access-2zl8h") pod "fc7eeac0-8c21-49d3-b9f1-43f850df2cea" (UID: "fc7eeac0-8c21-49d3-b9f1-43f850df2cea"). InnerVolumeSpecName "kube-api-access-2zl8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:18:50.998255 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:50.998163 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6973da3-09ec-4aa2-a0dd-e525a8289ca8-kube-api-access-9qsj9" (OuterVolumeSpecName: "kube-api-access-9qsj9") pod "b6973da3-09ec-4aa2-a0dd-e525a8289ca8" (UID: "b6973da3-09ec-4aa2-a0dd-e525a8289ca8"). InnerVolumeSpecName "kube-api-access-9qsj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:18:51.046270 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:51.046243 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0f3e-fz6dp"] Sep 08 04:18:51.051238 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:51.051219 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0f3e-fz6dp"] Sep 08 04:18:51.097297 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:51.097221 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6973da3-09ec-4aa2-a0dd-e525a8289ca8-plugins-config-volume\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:18:51.097297 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:51.097254 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qsj9\" (UniqueName: \"kubernetes.io/projected/b6973da3-09ec-4aa2-a0dd-e525a8289ca8-kube-api-access-9qsj9\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:18:51.097297 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:51.097264 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zl8h\" (UniqueName: \"kubernetes.io/projected/fc7eeac0-8c21-49d3-b9f1-43f850df2cea-kube-api-access-2zl8h\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:18:51.097297 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:51.097273 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7eeac0-8c21-49d3-b9f1-43f850df2cea-plugins-config-volume\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:18:51.109067 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:51.109038 2574 generic.go:356] "Generic (PLEG): container finished" podID="fc7eeac0-8c21-49d3-b9f1-43f850df2cea" containerID="e74babdef86ee92638a9c996d22609a7bed987f77ef671f2ea0d1e9d9bbae91c" exitCode=0 Sep 08 04:18:51.109160 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:51.109072 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7" event={"ID":"fc7eeac0-8c21-49d3-b9f1-43f850df2cea","Type":"ContainerDied","Data":"e74babdef86ee92638a9c996d22609a7bed987f77ef671f2ea0d1e9d9bbae91c"} Sep 08 04:18:51.109160 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:51.109088 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7" Sep 08 04:18:51.109160 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:51.109130 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7" event={"ID":"fc7eeac0-8c21-49d3-b9f1-43f850df2cea","Type":"ContainerDied","Data":"eef7522e8bd8674fbcf0688437a9ed19018d135caf8da785de51b9815935409d"} Sep 08 04:18:51.109160 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:51.109154 2574 scope.go:117] "RemoveContainer" containerID="e74babdef86ee92638a9c996d22609a7bed987f77ef671f2ea0d1e9d9bbae91c" Sep 08 04:18:51.110598 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:51.110574 2574 generic.go:356] "Generic (PLEG): container finished" podID="b6973da3-09ec-4aa2-a0dd-e525a8289ca8" containerID="90e3f0910ea20d2df22fa5da19b54c144c1c8906f17ec3e0d775a31646291598" exitCode=0 Sep 08 04:18:51.110762 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:51.110597 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c" event={"ID":"b6973da3-09ec-4aa2-a0dd-e525a8289ca8","Type":"ContainerDied","Data":"90e3f0910ea20d2df22fa5da19b54c144c1c8906f17ec3e0d775a31646291598"} Sep 08 04:18:51.110762 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:51.110625 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c" Sep 08 04:18:51.110762 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:51.110643 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c" event={"ID":"b6973da3-09ec-4aa2-a0dd-e525a8289ca8","Type":"ContainerDied","Data":"f957aaa1720c2da9ea909d28a8344afd69871736fb1e00a5ad282db48ac917e9"} Sep 08 04:18:51.120042 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:51.120025 2574 scope.go:117] "RemoveContainer" containerID="e74babdef86ee92638a9c996d22609a7bed987f77ef671f2ea0d1e9d9bbae91c" Sep 08 04:18:51.120375 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:18:51.120347 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e74babdef86ee92638a9c996d22609a7bed987f77ef671f2ea0d1e9d9bbae91c\": container with ID starting with e74babdef86ee92638a9c996d22609a7bed987f77ef671f2ea0d1e9d9bbae91c not found: ID does not exist" containerID="e74babdef86ee92638a9c996d22609a7bed987f77ef671f2ea0d1e9d9bbae91c" Sep 08 04:18:51.120465 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:51.120386 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e74babdef86ee92638a9c996d22609a7bed987f77ef671f2ea0d1e9d9bbae91c"} err="failed to get container status \"e74babdef86ee92638a9c996d22609a7bed987f77ef671f2ea0d1e9d9bbae91c\": rpc error: code = NotFound desc = could not find container \"e74babdef86ee92638a9c996d22609a7bed987f77ef671f2ea0d1e9d9bbae91c\": container with ID starting with e74babdef86ee92638a9c996d22609a7bed987f77ef671f2ea0d1e9d9bbae91c not found: ID does not exist" Sep 08 04:18:51.120465 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:51.120410 2574 scope.go:117] "RemoveContainer" containerID="90e3f0910ea20d2df22fa5da19b54c144c1c8906f17ec3e0d775a31646291598" Sep 08 04:18:51.122814 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:51.122793 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:51.128178 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:51.128164 2574 scope.go:117] "RemoveContainer" containerID="90e3f0910ea20d2df22fa5da19b54c144c1c8906f17ec3e0d775a31646291598" Sep 08 04:18:51.128424 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:18:51.128404 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90e3f0910ea20d2df22fa5da19b54c144c1c8906f17ec3e0d775a31646291598\": container with ID starting with 90e3f0910ea20d2df22fa5da19b54c144c1c8906f17ec3e0d775a31646291598 not found: ID does not exist" containerID="90e3f0910ea20d2df22fa5da19b54c144c1c8906f17ec3e0d775a31646291598" Sep 08 04:18:51.128482 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:51.128434 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e3f0910ea20d2df22fa5da19b54c144c1c8906f17ec3e0d775a31646291598"} err="failed to get container status \"90e3f0910ea20d2df22fa5da19b54c144c1c8906f17ec3e0d775a31646291598\": rpc error: code = NotFound desc = could not find container \"90e3f0910ea20d2df22fa5da19b54c144c1c8906f17ec3e0d775a31646291598\": container with ID starting with 90e3f0910ea20d2df22fa5da19b54c144c1c8906f17ec3e0d775a31646291598 not found: ID does not exist" Sep 08 04:18:51.153244 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:51.153213 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7"] Sep 08 04:18:51.168725 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:51.168700 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ff0f3e-5f56dc657-sshf7"] Sep 08 04:18:51.199819 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:51.199799 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c"] Sep 08 04:18:51.214193 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:51.214172 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ff0f3e-59d579689d-6t54c"] Sep 08 04:18:51.429473 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:51.429416 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:52.568406 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:52.568364 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3791a8ca-9381-49f8-96a0-314431703eb7" path="/var/lib/kubelet/pods/3791a8ca-9381-49f8-96a0-314431703eb7/volumes" Sep 08 04:18:52.568812 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:52.568783 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6973da3-09ec-4aa2-a0dd-e525a8289ca8" path="/var/lib/kubelet/pods/b6973da3-09ec-4aa2-a0dd-e525a8289ca8/volumes" Sep 08 04:18:52.569134 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:52.569117 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc7eeac0-8c21-49d3-b9f1-43f850df2cea" path="/var/lib/kubelet/pods/fc7eeac0-8c21-49d3-b9f1-43f850df2cea/volumes" Sep 08 04:18:55.301427 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:55.301391 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:55.415602 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:55.415570 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:55.706496 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:55.706427 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:56.075701 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:56.075672 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:56.252620 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:56.252583 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:56.433260 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:56.433179 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:18:59.923235 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:18:59.923202 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:00.062202 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.062170 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:00.219192 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.219117 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75"] Sep 08 04:19:00.219350 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.219328 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" podUID="4517c002-a5f8-4b0f-9002-ec403643ac9f" containerName="istio-proxy" containerID="cri-o://57ce77fc327c64d444745bbafbc897a5e2d282cd19927ab78b03a314cc8fb593" gracePeriod=30 Sep 08 04:19:00.305898 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.305869 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:00.612337 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.612303 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf5ae2-hv4vt"] Sep 08 04:19:00.612643 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.612630 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="583e207e-351b-438a-b200-b5536c3d6fcb" containerName="payload-pre-processing" Sep 08 04:19:00.612695 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.612644 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="583e207e-351b-438a-b200-b5536c3d6fcb" containerName="payload-pre-processing" Sep 08 04:19:00.612695 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.612656 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc7eeac0-8c21-49d3-b9f1-43f850df2cea" containerName="payload-processing" Sep 08 04:19:00.612695 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.612661 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc7eeac0-8c21-49d3-b9f1-43f850df2cea" containerName="payload-processing" Sep 08 04:19:00.612695 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.612673 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3791a8ca-9381-49f8-96a0-314431703eb7" containerName="revoke-keys" Sep 08 04:19:00.612695 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.612678 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3791a8ca-9381-49f8-96a0-314431703eb7" containerName="revoke-keys" Sep 08 04:19:00.612695 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.612689 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fbb39c6-6afc-478b-8db9-0b31818ee42e" containerName="payload-processing" Sep 08 04:19:00.612695 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.612695 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fbb39c6-6afc-478b-8db9-0b31818ee42e" containerName="payload-processing" Sep 08 04:19:00.612907 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.612706 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6973da3-09ec-4aa2-a0dd-e525a8289ca8" containerName="payload-pre-processing" Sep 08 04:19:00.612907 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.612712 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6973da3-09ec-4aa2-a0dd-e525a8289ca8" containerName="payload-pre-processing" Sep 08 04:19:00.612907 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.612761 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc7eeac0-8c21-49d3-b9f1-43f850df2cea" containerName="payload-processing" Sep 08 04:19:00.612907 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.612771 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="583e207e-351b-438a-b200-b5536c3d6fcb" containerName="payload-pre-processing" Sep 08 04:19:00.612907 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.612777 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fbb39c6-6afc-478b-8db9-0b31818ee42e" containerName="payload-processing" Sep 08 04:19:00.612907 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.612784 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3791a8ca-9381-49f8-96a0-314431703eb7" containerName="revoke-keys" Sep 08 04:19:00.612907 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.612789 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6973da3-09ec-4aa2-a0dd-e525a8289ca8" containerName="payload-pre-processing" Sep 08 04:19:00.615801 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.615784 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf5ae2-hv4vt" Sep 08 04:19:00.619499 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.619472 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7r29w\"" Sep 08 04:19:00.619677 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.619664 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 04:19:00.634151 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.634129 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf5ae2-hv4vt"] Sep 08 04:19:00.671598 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.671571 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22d8fb1f-307a-46a9-9ddd-fb2201e6b824-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cf5ae2-hv4vt\" (UID: \"22d8fb1f-307a-46a9-9ddd-fb2201e6b824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf5ae2-hv4vt" Sep 08 04:19:00.772558 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.772522 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22d8fb1f-307a-46a9-9ddd-fb2201e6b824-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cf5ae2-hv4vt\" (UID: \"22d8fb1f-307a-46a9-9ddd-fb2201e6b824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf5ae2-hv4vt" Sep 08 04:19:00.773130 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.773105 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22d8fb1f-307a-46a9-9ddd-fb2201e6b824-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cf5ae2-hv4vt\" (UID: \"22d8fb1f-307a-46a9-9ddd-fb2201e6b824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf5ae2-hv4vt" Sep 08 04:19:00.913716 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.913632 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:00.926490 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:00.926463 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf5ae2-hv4vt" Sep 08 04:19:01.221571 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:19:01.221540 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d8fb1f_307a_46a9_9ddd_fb2201e6b824.slice/crio-2a6964b501729409f846ca8b4b7aaf459f2fe1a77fcd4bceacc2d795924bac0b WatchSource:0}: Error finding container 2a6964b501729409f846ca8b4b7aaf459f2fe1a77fcd4bceacc2d795924bac0b: Status 404 returned error can't find the container with id 2a6964b501729409f846ca8b4b7aaf459f2fe1a77fcd4bceacc2d795924bac0b Sep 08 04:19:01.222958 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:01.222936 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf5ae2-hv4vt"] Sep 08 04:19:02.179912 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:02.179844 2574 generic.go:356] "Generic (PLEG): container finished" podID="22d8fb1f-307a-46a9-9ddd-fb2201e6b824" containerID="1c04b92941c729acd03937c1565991c39f9e9e78f794a26df8707e253434df1d" exitCode=0 Sep 08 04:19:02.180218 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:02.179936 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf5ae2-hv4vt" event={"ID":"22d8fb1f-307a-46a9-9ddd-fb2201e6b824","Type":"ContainerDied","Data":"1c04b92941c729acd03937c1565991c39f9e9e78f794a26df8707e253434df1d"} Sep 08 04:19:02.180218 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:02.179990 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf5ae2-hv4vt" event={"ID":"22d8fb1f-307a-46a9-9ddd-fb2201e6b824","Type":"ContainerStarted","Data":"2a6964b501729409f846ca8b4b7aaf459f2fe1a77fcd4bceacc2d795924bac0b"} Sep 08 04:19:03.317323 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:03.317300 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf5ae2-hv4vt" Sep 08 04:19:03.394551 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:03.394529 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22d8fb1f-307a-46a9-9ddd-fb2201e6b824-maas-api-service-ca\") pod \"22d8fb1f-307a-46a9-9ddd-fb2201e6b824\" (UID: \"22d8fb1f-307a-46a9-9ddd-fb2201e6b824\") " Sep 08 04:19:03.394816 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:03.394795 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22d8fb1f-307a-46a9-9ddd-fb2201e6b824-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "22d8fb1f-307a-46a9-9ddd-fb2201e6b824" (UID: "22d8fb1f-307a-46a9-9ddd-fb2201e6b824"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:19:03.495272 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:03.495218 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22d8fb1f-307a-46a9-9ddd-fb2201e6b824-maas-api-service-ca\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:19:03.920207 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:03.920175 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:04.098154 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:04.098117 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:04.192419 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:04.192356 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf5ae2-hv4vt" Sep 08 04:19:04.192419 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:04.192403 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf5ae2-hv4vt" event={"ID":"22d8fb1f-307a-46a9-9ddd-fb2201e6b824","Type":"ContainerDied","Data":"2a6964b501729409f846ca8b4b7aaf459f2fe1a77fcd4bceacc2d795924bac0b"} Sep 08 04:19:04.192597 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:04.192449 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a6964b501729409f846ca8b4b7aaf459f2fe1a77fcd4bceacc2d795924bac0b" Sep 08 04:19:04.252057 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:04.252018 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:04.412723 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:04.412696 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:05.473171 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.473150 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:19:05.614337 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.614309 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4517c002-a5f8-4b0f-9002-ec403643ac9f-istio-podinfo\") pod \"4517c002-a5f8-4b0f-9002-ec403643ac9f\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " Sep 08 04:19:05.614463 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.614344 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-istio-envoy\") pod \"4517c002-a5f8-4b0f-9002-ec403643ac9f\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " Sep 08 04:19:05.614463 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.614372 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4517c002-a5f8-4b0f-9002-ec403643ac9f-istio-token\") pod \"4517c002-a5f8-4b0f-9002-ec403643ac9f\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " Sep 08 04:19:05.614463 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.614396 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-workload-socket\") pod \"4517c002-a5f8-4b0f-9002-ec403643ac9f\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " Sep 08 04:19:05.614463 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.614430 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4517c002-a5f8-4b0f-9002-ec403643ac9f-istiod-ca-cert\") pod \"4517c002-a5f8-4b0f-9002-ec403643ac9f\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " Sep 08 04:19:05.614463 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.614448 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48vpp\" (UniqueName: \"kubernetes.io/projected/4517c002-a5f8-4b0f-9002-ec403643ac9f-kube-api-access-48vpp\") pod \"4517c002-a5f8-4b0f-9002-ec403643ac9f\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " Sep 08 04:19:05.614753 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.614482 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-credential-socket\") pod \"4517c002-a5f8-4b0f-9002-ec403643ac9f\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " Sep 08 04:19:05.614753 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.614553 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-workload-certs\") pod \"4517c002-a5f8-4b0f-9002-ec403643ac9f\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " Sep 08 04:19:05.614753 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.614590 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-istio-data\") pod \"4517c002-a5f8-4b0f-9002-ec403643ac9f\" (UID: \"4517c002-a5f8-4b0f-9002-ec403643ac9f\") " Sep 08 04:19:05.614917 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.614797 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4517c002-a5f8-4b0f-9002-ec403643ac9f" (UID: "4517c002-a5f8-4b0f-9002-ec403643ac9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:19:05.614917 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.614886 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-workload-socket\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:19:05.615032 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.614905 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4517c002-a5f8-4b0f-9002-ec403643ac9f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4517c002-a5f8-4b0f-9002-ec403643ac9f" (UID: "4517c002-a5f8-4b0f-9002-ec403643ac9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:19:05.615106 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.615083 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4517c002-a5f8-4b0f-9002-ec403643ac9f" (UID: "4517c002-a5f8-4b0f-9002-ec403643ac9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:19:05.615205 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.615184 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4517c002-a5f8-4b0f-9002-ec403643ac9f" (UID: "4517c002-a5f8-4b0f-9002-ec403643ac9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:19:05.615266 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.615236 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-istio-data" (OuterVolumeSpecName: "istio-data") pod "4517c002-a5f8-4b0f-9002-ec403643ac9f" (UID: "4517c002-a5f8-4b0f-9002-ec403643ac9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:19:05.616861 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.616832 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4517c002-a5f8-4b0f-9002-ec403643ac9f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4517c002-a5f8-4b0f-9002-ec403643ac9f" (UID: "4517c002-a5f8-4b0f-9002-ec403643ac9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 04:19:05.616981 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.616860 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4517c002-a5f8-4b0f-9002-ec403643ac9f-istio-token" (OuterVolumeSpecName: "istio-token") pod "4517c002-a5f8-4b0f-9002-ec403643ac9f" (UID: "4517c002-a5f8-4b0f-9002-ec403643ac9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:19:05.617106 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.617080 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4517c002-a5f8-4b0f-9002-ec403643ac9f" (UID: "4517c002-a5f8-4b0f-9002-ec403643ac9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:19:05.617205 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.617145 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4517c002-a5f8-4b0f-9002-ec403643ac9f-kube-api-access-48vpp" (OuterVolumeSpecName: "kube-api-access-48vpp") pod "4517c002-a5f8-4b0f-9002-ec403643ac9f" (UID: "4517c002-a5f8-4b0f-9002-ec403643ac9f"). InnerVolumeSpecName "kube-api-access-48vpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:19:05.716170 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.716148 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4517c002-a5f8-4b0f-9002-ec403643ac9f-istio-token\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:19:05.716170 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.716168 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4517c002-a5f8-4b0f-9002-ec403643ac9f-istiod-ca-cert\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:19:05.716294 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.716178 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48vpp\" (UniqueName: \"kubernetes.io/projected/4517c002-a5f8-4b0f-9002-ec403643ac9f-kube-api-access-48vpp\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:19:05.716294 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.716187 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-credential-socket\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:19:05.716294 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.716196 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-workload-certs\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:19:05.716294 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.716205 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-istio-data\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:19:05.716294 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.716212 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4517c002-a5f8-4b0f-9002-ec403643ac9f-istio-podinfo\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:19:05.716294 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:05.716220 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4517c002-a5f8-4b0f-9002-ec403643ac9f-istio-envoy\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:19:06.213135 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:06.213109 2574 generic.go:356] "Generic (PLEG): container finished" podID="4517c002-a5f8-4b0f-9002-ec403643ac9f" containerID="57ce77fc327c64d444745bbafbc897a5e2d282cd19927ab78b03a314cc8fb593" exitCode=0 Sep 08 04:19:06.213299 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:06.213158 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" Sep 08 04:19:06.213299 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:06.213181 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" event={"ID":"4517c002-a5f8-4b0f-9002-ec403643ac9f","Type":"ContainerDied","Data":"57ce77fc327c64d444745bbafbc897a5e2d282cd19927ab78b03a314cc8fb593"} Sep 08 04:19:06.213299 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:06.213257 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75" event={"ID":"4517c002-a5f8-4b0f-9002-ec403643ac9f","Type":"ContainerDied","Data":"0c91f67f83b97d5716c41b2a4f98fbd09a8c797098ab9725095e9e4d5a513fd1"} Sep 08 04:19:06.213299 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:06.213271 2574 scope.go:117] "RemoveContainer" containerID="57ce77fc327c64d444745bbafbc897a5e2d282cd19927ab78b03a314cc8fb593" Sep 08 04:19:06.223608 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:06.223592 2574 scope.go:117] "RemoveContainer" containerID="e126d8fa775f77354bb1c36c2fc3d0482bd9c259707d5a14233481f386584db1" Sep 08 04:19:06.231800 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:06.231784 2574 scope.go:117] "RemoveContainer" containerID="57ce77fc327c64d444745bbafbc897a5e2d282cd19927ab78b03a314cc8fb593" Sep 08 04:19:06.232026 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:19:06.232007 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57ce77fc327c64d444745bbafbc897a5e2d282cd19927ab78b03a314cc8fb593\": container with ID starting with 57ce77fc327c64d444745bbafbc897a5e2d282cd19927ab78b03a314cc8fb593 not found: ID does not exist" containerID="57ce77fc327c64d444745bbafbc897a5e2d282cd19927ab78b03a314cc8fb593" Sep 08 04:19:06.232067 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:06.232033 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57ce77fc327c64d444745bbafbc897a5e2d282cd19927ab78b03a314cc8fb593"} err="failed to get container status \"57ce77fc327c64d444745bbafbc897a5e2d282cd19927ab78b03a314cc8fb593\": rpc error: code = NotFound desc = could not find container \"57ce77fc327c64d444745bbafbc897a5e2d282cd19927ab78b03a314cc8fb593\": container with ID starting with 57ce77fc327c64d444745bbafbc897a5e2d282cd19927ab78b03a314cc8fb593 not found: ID does not exist" Sep 08 04:19:06.232067 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:06.232051 2574 scope.go:117] "RemoveContainer" containerID="e126d8fa775f77354bb1c36c2fc3d0482bd9c259707d5a14233481f386584db1" Sep 08 04:19:06.232329 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:19:06.232308 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e126d8fa775f77354bb1c36c2fc3d0482bd9c259707d5a14233481f386584db1\": container with ID starting with e126d8fa775f77354bb1c36c2fc3d0482bd9c259707d5a14233481f386584db1 not found: ID does not exist" containerID="e126d8fa775f77354bb1c36c2fc3d0482bd9c259707d5a14233481f386584db1" Sep 08 04:19:06.232407 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:06.232335 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e126d8fa775f77354bb1c36c2fc3d0482bd9c259707d5a14233481f386584db1"} err="failed to get container status \"e126d8fa775f77354bb1c36c2fc3d0482bd9c259707d5a14233481f386584db1\": rpc error: code = NotFound desc = could not find container \"e126d8fa775f77354bb1c36c2fc3d0482bd9c259707d5a14233481f386584db1\": container with ID starting with e126d8fa775f77354bb1c36c2fc3d0482bd9c259707d5a14233481f386584db1 not found: ID does not exist" Sep 08 04:19:06.249888 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:06.249864 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75"] Sep 08 04:19:06.264684 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:06.264664 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ff0f3e-openshift-default-7cd7457974-c9j75"] Sep 08 04:19:06.567636 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:06.567600 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4517c002-a5f8-4b0f-9002-ec403643ac9f" path="/var/lib/kubelet/pods/4517c002-a5f8-4b0f-9002-ec403643ac9f/volumes" Sep 08 04:19:09.951520 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:09.951481 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:11.940940 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:11.940900 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:12.217787 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:12.217710 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:12.394835 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:12.394799 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:12.478244 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:12.478168 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf5ae2-hv4vt"] Sep 08 04:19:12.489209 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:12.489182 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf5ae2-hv4vt"] Sep 08 04:19:12.567794 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:12.567759 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22d8fb1f-307a-46a9-9ddd-fb2201e6b824" path="/var/lib/kubelet/pods/22d8fb1f-307a-46a9-9ddd-fb2201e6b824/volumes" Sep 08 04:19:12.590987 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:12.590962 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:12.918295 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:12.918270 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:13.309001 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:13.308976 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:13.513954 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:13.513926 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:14.360650 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.360615 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:14.781764 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.781729 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf"] Sep 08 04:19:14.782174 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.782148 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4517c002-a5f8-4b0f-9002-ec403643ac9f" containerName="istio-proxy" Sep 08 04:19:14.782174 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.782175 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4517c002-a5f8-4b0f-9002-ec403643ac9f" containerName="istio-proxy" Sep 08 04:19:14.782336 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.782192 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22d8fb1f-307a-46a9-9ddd-fb2201e6b824" containerName="revoke-keys" Sep 08 04:19:14.782336 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.782217 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="22d8fb1f-307a-46a9-9ddd-fb2201e6b824" containerName="revoke-keys" Sep 08 04:19:14.782336 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.782228 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4517c002-a5f8-4b0f-9002-ec403643ac9f" containerName="istio-proxy" Sep 08 04:19:14.782336 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.782235 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4517c002-a5f8-4b0f-9002-ec403643ac9f" containerName="istio-proxy" Sep 08 04:19:14.782336 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.782298 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4517c002-a5f8-4b0f-9002-ec403643ac9f" containerName="istio-proxy" Sep 08 04:19:14.782336 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.782306 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4517c002-a5f8-4b0f-9002-ec403643ac9f" containerName="istio-proxy" Sep 08 04:19:14.782336 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.782314 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="22d8fb1f-307a-46a9-9ddd-fb2201e6b824" containerName="revoke-keys" Sep 08 04:19:14.786673 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.786657 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:14.790212 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.790183 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-75264d-openshift-default-dockercfg-6tttg\"" Sep 08 04:19:14.821832 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.821811 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:14.824230 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.824207 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf"] Sep 08 04:19:14.881159 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.881129 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-workload-certs\") pod \"e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:14.881302 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.881179 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-istio-envoy\") pod \"e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:14.881302 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.881200 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/078876db-1081-4a8a-bbc4-278589a1f9ad-istiod-ca-cert\") pod \"e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:14.881302 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.881270 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-credential-socket\") pod \"e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:14.881415 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.881325 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-istio-data\") pod \"e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:14.881415 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.881344 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-workload-socket\") pod \"e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:14.881415 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.881361 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/078876db-1081-4a8a-bbc4-278589a1f9ad-istio-token\") pod \"e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:14.881415 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.881378 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7jzn\" (UniqueName: \"kubernetes.io/projected/078876db-1081-4a8a-bbc4-278589a1f9ad-kube-api-access-s7jzn\") pod \"e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:14.881578 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.881437 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/078876db-1081-4a8a-bbc4-278589a1f9ad-istio-podinfo\") pod \"e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:14.982830 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.982803 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-workload-certs\") pod \"e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:14.982945 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.982844 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-istio-envoy\") pod \"e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:14.982945 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.982859 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/078876db-1081-4a8a-bbc4-278589a1f9ad-istiod-ca-cert\") pod \"e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:14.982945 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.982877 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-credential-socket\") pod \"e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:14.982945 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.982905 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-istio-data\") pod \"e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:14.982945 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.982931 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-workload-socket\") pod \"e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:14.983204 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.982951 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/078876db-1081-4a8a-bbc4-278589a1f9ad-istio-token\") pod \"e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:14.983204 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.982968 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7jzn\" (UniqueName: \"kubernetes.io/projected/078876db-1081-4a8a-bbc4-278589a1f9ad-kube-api-access-s7jzn\") pod \"e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:14.983204 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.983008 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/078876db-1081-4a8a-bbc4-278589a1f9ad-istio-podinfo\") pod \"e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:14.983351 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.983252 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-workload-certs\") pod \"e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:14.983490 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.983461 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-istio-data\") pod \"e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:14.983637 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.983530 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-credential-socket\") pod \"e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:14.983637 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.983564 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-workload-socket\") pod \"e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:14.983637 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.983620 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/078876db-1081-4a8a-bbc4-278589a1f9ad-istiod-ca-cert\") pod \"e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:14.985252 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.985225 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-istio-envoy\") pod \"e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:14.985608 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:14.985590 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/078876db-1081-4a8a-bbc4-278589a1f9ad-istio-podinfo\") pod \"e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:15.005975 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:15.005954 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7jzn\" (UniqueName: \"kubernetes.io/projected/078876db-1081-4a8a-bbc4-278589a1f9ad-kube-api-access-s7jzn\") pod \"e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:15.027325 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:15.027300 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/078876db-1081-4a8a-bbc4-278589a1f9ad-istio-token\") pod \"e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:15.096467 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:15.096409 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:15.110139 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:15.110111 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:15.410970 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:15.410883 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf"] Sep 08 04:19:15.416124 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:19:15.416098 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078876db_1081_4a8a_bbc4_278589a1f9ad.slice/crio-4d7b30e3eb7924d3e8b353f279a32d05ae6448a7592a151fe330cfbd6988f668 WatchSource:0}: Error finding container 4d7b30e3eb7924d3e8b353f279a32d05ae6448a7592a151fe330cfbd6988f668: Status 404 returned error can't find the container with id 4d7b30e3eb7924d3e8b353f279a32d05ae6448a7592a151fe330cfbd6988f668 Sep 08 04:19:15.418210 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:15.418172 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:19:15.418304 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:15.418236 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:19:15.418304 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:15.418267 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:19:16.280337 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:16.280302 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" event={"ID":"078876db-1081-4a8a-bbc4-278589a1f9ad","Type":"ContainerStarted","Data":"8f5126313d4618c4d9615a0c2ec95a43c1c83bf9cf176522a465efd7c925e7c2"} Sep 08 04:19:16.280553 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:16.280352 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" event={"ID":"078876db-1081-4a8a-bbc4-278589a1f9ad","Type":"ContainerStarted","Data":"4d7b30e3eb7924d3e8b353f279a32d05ae6448a7592a151fe330cfbd6988f668"} Sep 08 04:19:16.328832 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:16.328783 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" podStartSLOduration=2.328767542 podStartE2EDuration="2.328767542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:19:16.324848548 +0000 UTC m=+1130.247309308" watchObservedRunningTime="2026-09-08 04:19:16.328767542 +0000 UTC m=+1130.251228333" Sep 08 04:19:17.096849 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:17.096813 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:17.102318 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:17.102288 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:17.286966 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:17.286931 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:17.287856 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:17.287834 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:18.666060 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:18.666023 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:18.810342 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:18.810312 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:20.065253 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:20.065208 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:20.264870 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:20.264834 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:20.425606 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:20.425532 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:20.992603 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:20.992564 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:21.146295 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:21.146266 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:21.218066 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:21.218034 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw"] Sep 08 04:19:21.221633 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:21.221617 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw" Sep 08 04:19:21.227679 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:21.227655 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-75264d-dockercfg-g2nlq\"" Sep 08 04:19:21.228171 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:21.228152 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-75264d\"" Sep 08 04:19:21.250155 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:21.250128 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw"] Sep 08 04:19:21.341988 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:21.341959 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39d4d068-2ae5-4796-9399-8a2cb889d7e7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw\" (UID: \"39d4d068-2ae5-4796-9399-8a2cb889d7e7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw" Sep 08 04:19:21.342118 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:21.342027 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kgr6\" (UniqueName: \"kubernetes.io/projected/39d4d068-2ae5-4796-9399-8a2cb889d7e7-kube-api-access-6kgr6\") pod \"payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw\" (UID: \"39d4d068-2ae5-4796-9399-8a2cb889d7e7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw" Sep 08 04:19:21.362324 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:21.362290 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz"] Sep 08 04:19:21.365713 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:21.365695 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz" Sep 08 04:19:21.389690 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:21.389668 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz"] Sep 08 04:19:21.443335 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:21.443313 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39d4d068-2ae5-4796-9399-8a2cb889d7e7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw\" (UID: \"39d4d068-2ae5-4796-9399-8a2cb889d7e7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw" Sep 08 04:19:21.443434 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:21.443373 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kgr6\" (UniqueName: \"kubernetes.io/projected/39d4d068-2ae5-4796-9399-8a2cb889d7e7-kube-api-access-6kgr6\") pod \"payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw\" (UID: \"39d4d068-2ae5-4796-9399-8a2cb889d7e7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw" Sep 08 04:19:21.443995 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:21.443978 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39d4d068-2ae5-4796-9399-8a2cb889d7e7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw\" (UID: \"39d4d068-2ae5-4796-9399-8a2cb889d7e7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw" Sep 08 04:19:21.456386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:21.456362 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kgr6\" (UniqueName: \"kubernetes.io/projected/39d4d068-2ae5-4796-9399-8a2cb889d7e7-kube-api-access-6kgr6\") pod \"payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw\" (UID: \"39d4d068-2ae5-4796-9399-8a2cb889d7e7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw" Sep 08 04:19:21.531854 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:21.531804 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw" Sep 08 04:19:21.544070 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:21.544047 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2f2q\" (UniqueName: \"kubernetes.io/projected/07080ec2-37dc-425a-b8e4-d4843da3250d-kube-api-access-j2f2q\") pod \"payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz\" (UID: \"07080ec2-37dc-425a-b8e4-d4843da3250d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz" Sep 08 04:19:21.544144 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:21.544132 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07080ec2-37dc-425a-b8e4-d4843da3250d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz\" (UID: \"07080ec2-37dc-425a-b8e4-d4843da3250d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz" Sep 08 04:19:21.644795 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:21.644758 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07080ec2-37dc-425a-b8e4-d4843da3250d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz\" (UID: \"07080ec2-37dc-425a-b8e4-d4843da3250d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz" Sep 08 04:19:21.644940 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:21.644851 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2f2q\" (UniqueName: \"kubernetes.io/projected/07080ec2-37dc-425a-b8e4-d4843da3250d-kube-api-access-j2f2q\") pod \"payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz\" (UID: \"07080ec2-37dc-425a-b8e4-d4843da3250d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz" Sep 08 04:19:21.645448 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:21.645425 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07080ec2-37dc-425a-b8e4-d4843da3250d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz\" (UID: \"07080ec2-37dc-425a-b8e4-d4843da3250d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz" Sep 08 04:19:21.659105 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:21.659078 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2f2q\" (UniqueName: \"kubernetes.io/projected/07080ec2-37dc-425a-b8e4-d4843da3250d-kube-api-access-j2f2q\") pod \"payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz\" (UID: \"07080ec2-37dc-425a-b8e4-d4843da3250d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz" Sep 08 04:19:21.675951 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:21.675931 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz" Sep 08 04:19:21.823829 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:21.823799 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw"] Sep 08 04:19:21.824755 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:19:21.824731 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d4d068_2ae5_4796_9399_8a2cb889d7e7.slice/crio-c658ac88c15d1e7347bcb769a9fc98125c44d4d1ccd587ad107427bf7d2db363 WatchSource:0}: Error finding container c658ac88c15d1e7347bcb769a9fc98125c44d4d1ccd587ad107427bf7d2db363: Status 404 returned error can't find the container with id c658ac88c15d1e7347bcb769a9fc98125c44d4d1ccd587ad107427bf7d2db363 Sep 08 04:19:22.025423 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:19:22.025393 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07080ec2_37dc_425a_b8e4_d4843da3250d.slice/crio-8ae46eb572c7e5930a78a8b8184a39e465f34d715b2f5a842e297b64b6a97486 WatchSource:0}: Error finding container 8ae46eb572c7e5930a78a8b8184a39e465f34d715b2f5a842e297b64b6a97486: Status 404 returned error can't find the container with id 8ae46eb572c7e5930a78a8b8184a39e465f34d715b2f5a842e297b64b6a97486 Sep 08 04:19:22.033381 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:22.033360 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz"] Sep 08 04:19:22.326925 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:22.326842 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw" event={"ID":"39d4d068-2ae5-4796-9399-8a2cb889d7e7","Type":"ContainerStarted","Data":"9dac01312ba5657e083cfaae106140525f9c1c707efc3934530d569b988b6e6b"} Sep 08 04:19:22.327357 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:22.326927 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw" event={"ID":"39d4d068-2ae5-4796-9399-8a2cb889d7e7","Type":"ContainerStarted","Data":"c658ac88c15d1e7347bcb769a9fc98125c44d4d1ccd587ad107427bf7d2db363"} Sep 08 04:19:22.327357 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:22.326961 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw" Sep 08 04:19:22.328883 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:22.328845 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz" event={"ID":"07080ec2-37dc-425a-b8e4-d4843da3250d","Type":"ContainerStarted","Data":"bccae318b15a48c05d893c125900bd45e153fd8324992ea8ff8f36d5d20dc5b5"} Sep 08 04:19:22.329015 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:22.328897 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz" event={"ID":"07080ec2-37dc-425a-b8e4-d4843da3250d","Type":"ContainerStarted","Data":"8ae46eb572c7e5930a78a8b8184a39e465f34d715b2f5a842e297b64b6a97486"} Sep 08 04:19:22.329015 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:22.328942 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz" Sep 08 04:19:22.372103 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:22.372058 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw" podStartSLOduration=1.372043826 podStartE2EDuration="1.372043826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:19:22.36935737 +0000 UTC m=+1136.291818129" watchObservedRunningTime="2026-09-08 04:19:22.372043826 +0000 UTC m=+1136.294504566" Sep 08 04:19:22.426706 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:22.426657 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz" podStartSLOduration=1.426643809 podStartE2EDuration="1.426643809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:19:22.423372378 +0000 UTC m=+1136.345833154" watchObservedRunningTime="2026-09-08 04:19:22.426643809 +0000 UTC m=+1136.349104549" Sep 08 04:19:23.083000 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:23.082973 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:23.264965 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:23.264907 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:23.435574 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:23.435474 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:23.994521 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:23.994484 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:24.430586 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.430554 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt"] Sep 08 04:19:24.434066 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.434048 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.437268 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.437245 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-c2a8bb-openshift-default-dockercfg-pmksf\"" Sep 08 04:19:24.468770 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.468741 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt"] Sep 08 04:19:24.510063 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.510024 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:24.568682 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.568645 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc67f27e-765a-425a-9b20-d7d05f43301c-istio-token\") pod \"e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.568855 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.568702 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-istio-envoy\") pod \"e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.568855 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.568743 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-workload-socket\") pod \"e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.568855 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.568786 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc67f27e-765a-425a-9b20-d7d05f43301c-istio-podinfo\") pod \"e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.568855 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.568821 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-credential-socket\") pod \"e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.568855 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.568853 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljf6g\" (UniqueName: \"kubernetes.io/projected/fc67f27e-765a-425a-9b20-d7d05f43301c-kube-api-access-ljf6g\") pod \"e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.569175 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.568891 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-workload-certs\") pod \"e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.569175 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.568925 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-istio-data\") pod \"e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.569175 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.568980 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc67f27e-765a-425a-9b20-d7d05f43301c-istiod-ca-cert\") pod \"e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.656520 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.656485 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:24.670021 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.669991 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc67f27e-765a-425a-9b20-d7d05f43301c-istio-podinfo\") pod \"e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.670145 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.670034 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-credential-socket\") pod \"e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.670145 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.670056 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljf6g\" (UniqueName: \"kubernetes.io/projected/fc67f27e-765a-425a-9b20-d7d05f43301c-kube-api-access-ljf6g\") pod \"e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.670278 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.670217 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-workload-certs\") pod \"e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.670278 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.670260 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-istio-data\") pod \"e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.670389 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.670339 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc67f27e-765a-425a-9b20-d7d05f43301c-istiod-ca-cert\") pod \"e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.670445 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.670386 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc67f27e-765a-425a-9b20-d7d05f43301c-istio-token\") pod \"e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.670445 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.670420 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-istio-envoy\") pod \"e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.670581 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.670452 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-credential-socket\") pod \"e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.670581 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.670474 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-workload-socket\") pod \"e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.670992 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.670963 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-istio-data\") pod \"e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.671092 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.670995 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-workload-socket\") pod \"e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.671190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.671168 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-workload-certs\") pod \"e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.671602 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.671578 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc67f27e-765a-425a-9b20-d7d05f43301c-istiod-ca-cert\") pod \"e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.672822 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.672798 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-istio-envoy\") pod \"e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.672912 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.672893 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc67f27e-765a-425a-9b20-d7d05f43301c-istio-podinfo\") pod \"e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.685646 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.685596 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc67f27e-765a-425a-9b20-d7d05f43301c-istio-token\") pod \"e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.685775 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.685754 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljf6g\" (UniqueName: \"kubernetes.io/projected/fc67f27e-765a-425a-9b20-d7d05f43301c-kube-api-access-ljf6g\") pod \"e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:24.744628 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:24.744517 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:25.049482 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:25.049454 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt"] Sep 08 04:19:25.051463 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:19:25.051436 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc67f27e_765a_425a_9b20_d7d05f43301c.slice/crio-6fdcfb0f2c697b48441b64e7cda1529378859f6922b27c8650283a10f631732b WatchSource:0}: Error finding container 6fdcfb0f2c697b48441b64e7cda1529378859f6922b27c8650283a10f631732b: Status 404 returned error can't find the container with id 6fdcfb0f2c697b48441b64e7cda1529378859f6922b27c8650283a10f631732b Sep 08 04:19:25.053667 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:25.053637 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:19:25.053740 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:25.053701 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:19:25.053740 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:25.053733 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:19:25.350910 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:25.350814 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" event={"ID":"fc67f27e-765a-425a-9b20-d7d05f43301c","Type":"ContainerStarted","Data":"f2f7558d659683be33ebb0c84976d6c4088d8d6248c8076b1b9be94e8edfadac"} Sep 08 04:19:25.350910 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:25.350875 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" event={"ID":"fc67f27e-765a-425a-9b20-d7d05f43301c","Type":"ContainerStarted","Data":"6fdcfb0f2c697b48441b64e7cda1529378859f6922b27c8650283a10f631732b"} Sep 08 04:19:25.392291 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:25.392233 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" podStartSLOduration=1.39221752 podStartE2EDuration="1.39221752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:19:25.38999516 +0000 UTC m=+1139.312455920" watchObservedRunningTime="2026-09-08 04:19:25.39221752 +0000 UTC m=+1139.314678259" Sep 08 04:19:25.744896 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:25.744811 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:26.750486 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:26.750460 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:27.361956 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:27.361922 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:27.362902 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:27.362881 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:27.472023 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:27.471984 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:27.681885 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:27.681805 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:28.845875 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:28.845842 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:29.016321 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:29.016279 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:29.847974 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:29.847933 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:30.054370 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:30.054335 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:30.133068 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:30.132994 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8"] Sep 08 04:19:30.138118 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:30.138067 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8" Sep 08 04:19:30.142394 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:30.142375 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-c2a8bb-dockercfg-q7s9l\"" Sep 08 04:19:30.145780 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:30.145755 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-c2a8bb\"" Sep 08 04:19:30.155959 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:30.155933 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8"] Sep 08 04:19:30.229759 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:30.229731 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb7575b-685f-48e8-8616-b52b6722357f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8\" (UID: \"1eb7575b-685f-48e8-8616-b52b6722357f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8" Sep 08 04:19:30.229887 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:30.229785 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fngpz\" (UniqueName: \"kubernetes.io/projected/1eb7575b-685f-48e8-8616-b52b6722357f-kube-api-access-fngpz\") pod \"payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8\" (UID: \"1eb7575b-685f-48e8-8616-b52b6722357f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8" Sep 08 04:19:30.331137 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:30.331101 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb7575b-685f-48e8-8616-b52b6722357f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8\" (UID: \"1eb7575b-685f-48e8-8616-b52b6722357f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8" Sep 08 04:19:30.331301 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:30.331166 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fngpz\" (UniqueName: \"kubernetes.io/projected/1eb7575b-685f-48e8-8616-b52b6722357f-kube-api-access-fngpz\") pod \"payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8\" (UID: \"1eb7575b-685f-48e8-8616-b52b6722357f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8" Sep 08 04:19:30.331818 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:30.331798 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb7575b-685f-48e8-8616-b52b6722357f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8\" (UID: \"1eb7575b-685f-48e8-8616-b52b6722357f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8" Sep 08 04:19:30.359705 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:30.359675 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fngpz\" (UniqueName: \"kubernetes.io/projected/1eb7575b-685f-48e8-8616-b52b6722357f-kube-api-access-fngpz\") pod \"payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8\" (UID: \"1eb7575b-685f-48e8-8616-b52b6722357f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8" Sep 08 04:19:30.447979 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:30.447905 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8" Sep 08 04:19:30.754142 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:19:30.754117 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb7575b_685f_48e8_8616_b52b6722357f.slice/crio-941b882ffb152309148fe20324b80936c08e8af9b5672ab9d11d5f08b90657cc WatchSource:0}: Error finding container 941b882ffb152309148fe20324b80936c08e8af9b5672ab9d11d5f08b90657cc: Status 404 returned error can't find the container with id 941b882ffb152309148fe20324b80936c08e8af9b5672ab9d11d5f08b90657cc Sep 08 04:19:30.760398 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:30.760370 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8"] Sep 08 04:19:31.387066 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:31.386985 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8" event={"ID":"1eb7575b-685f-48e8-8616-b52b6722357f","Type":"ContainerStarted","Data":"61218c6991f09d802fbcad38744d56e24487b28d3c30a8758e336785a975b254"} Sep 08 04:19:31.387066 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:31.387032 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8" event={"ID":"1eb7575b-685f-48e8-8616-b52b6722357f","Type":"ContainerStarted","Data":"941b882ffb152309148fe20324b80936c08e8af9b5672ab9d11d5f08b90657cc"} Sep 08 04:19:31.387066 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:31.387063 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8" Sep 08 04:19:31.454876 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:31.454828 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8" podStartSLOduration=1.454814137 podStartE2EDuration="1.454814137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:19:31.452797528 +0000 UTC m=+1145.375258288" watchObservedRunningTime="2026-09-08 04:19:31.454814137 +0000 UTC m=+1145.377274876" Sep 08 04:19:33.328650 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:33.328606 2574 patch_prober.go:28] interesting pod/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 08 04:19:33.329003 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:33.328661 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" podUID="078876db-1081-4a8a-bbc4-278589a1f9ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 08 04:19:33.338939 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:33.338914 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz" Sep 08 04:19:33.339187 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:33.339168 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw" Sep 08 04:19:34.412258 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:34.412226 2574 generic.go:356] "Generic (PLEG): container finished" podID="078876db-1081-4a8a-bbc4-278589a1f9ad" containerID="8f5126313d4618c4d9615a0c2ec95a43c1c83bf9cf176522a465efd7c925e7c2" exitCode=0 Sep 08 04:19:34.412892 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:34.412293 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" event={"ID":"078876db-1081-4a8a-bbc4-278589a1f9ad","Type":"ContainerDied","Data":"8f5126313d4618c4d9615a0c2ec95a43c1c83bf9cf176522a465efd7c925e7c2"} Sep 08 04:19:34.413555 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:34.413538 2574 scope.go:117] "RemoveContainer" containerID="8f5126313d4618c4d9615a0c2ec95a43c1c83bf9cf176522a465efd7c925e7c2" Sep 08 04:19:34.415014 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:34.414976 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:19:34.415108 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:34.415090 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:19:34.415176 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:34.415145 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:19:35.423196 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:35.423160 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" event={"ID":"078876db-1081-4a8a-bbc4-278589a1f9ad","Type":"ContainerStarted","Data":"61af56d59574e2335f02e6aa4c6075554c59ff6ca4bed1c6f82b6786dfef23b0"} Sep 08 04:19:36.102528 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:36.101861 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:36.102528 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:36.102015 2574 patch_prober.go:28] interesting pod/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 08 04:19:36.102528 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:36.102061 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" podUID="078876db-1081-4a8a-bbc4-278589a1f9ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 08 04:19:37.097550 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:37.097486 2574 patch_prober.go:28] interesting pod/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 08 04:19:37.098011 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:37.097576 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" podUID="078876db-1081-4a8a-bbc4-278589a1f9ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 08 04:19:38.096807 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:38.096768 2574 patch_prober.go:28] interesting pod/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 08 04:19:38.097087 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:38.097058 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" podUID="078876db-1081-4a8a-bbc4-278589a1f9ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 08 04:19:38.964615 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:38.964587 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:39.096875 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:39.096839 2574 patch_prober.go:28] interesting pod/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 08 04:19:39.097170 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:39.097139 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" podUID="078876db-1081-4a8a-bbc4-278589a1f9ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 08 04:19:40.098046 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:40.097594 2574 patch_prober.go:28] interesting pod/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 08 04:19:40.098046 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:40.097659 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" podUID="078876db-1081-4a8a-bbc4-278589a1f9ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 08 04:19:41.140997 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:41.140947 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:41.141683 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:41.141649 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:41.142542 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:41.142496 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:19:41.945311 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:41.945261 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:42.396164 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:42.396132 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8" Sep 08 04:19:42.814789 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:42.814760 2574 patch_prober.go:28] interesting pod/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": read tcp 10.133.0.2:51586->10.133.0.73:15021: read: connection reset by peer" start-of-body= Sep 08 04:19:42.814967 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:42.814817 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" podUID="fc67f27e-765a-425a-9b20-d7d05f43301c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": read tcp 10.133.0.2:51586->10.133.0.73:15021: read: connection reset by peer" Sep 08 04:19:43.483960 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:43.483928 2574 generic.go:356] "Generic (PLEG): container finished" podID="fc67f27e-765a-425a-9b20-d7d05f43301c" containerID="f2f7558d659683be33ebb0c84976d6c4088d8d6248c8076b1b9be94e8edfadac" exitCode=0 Sep 08 04:19:43.483960 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:43.483966 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" event={"ID":"fc67f27e-765a-425a-9b20-d7d05f43301c","Type":"ContainerDied","Data":"f2f7558d659683be33ebb0c84976d6c4088d8d6248c8076b1b9be94e8edfadac"} Sep 08 04:19:43.484386 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:43.484321 2574 scope.go:117] "RemoveContainer" containerID="f2f7558d659683be33ebb0c84976d6c4088d8d6248c8076b1b9be94e8edfadac" Sep 08 04:19:43.485403 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:43.485372 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:19:43.485564 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:43.485475 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:19:43.485564 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:43.485548 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:19:43.624821 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:43.624789 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:43.915451 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:43.915353 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:44.414191 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:44.414151 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:44.492088 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:44.492054 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" event={"ID":"fc67f27e-765a-425a-9b20-d7d05f43301c","Type":"ContainerStarted","Data":"c7107cc712ca3f7e914a8470246491866ec982f8943fc4e004a5673cddba420b"} Sep 08 04:19:44.548210 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:44.548157 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:44.745404 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:44.745307 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:44.745636 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:44.745612 2574 patch_prober.go:28] interesting pod/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 08 04:19:44.745694 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:44.745662 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" podUID="fc67f27e-765a-425a-9b20-d7d05f43301c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 08 04:19:45.745202 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:45.745165 2574 patch_prober.go:28] interesting pod/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 08 04:19:45.745688 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:45.745224 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" podUID="fc67f27e-765a-425a-9b20-d7d05f43301c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 08 04:19:46.745440 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:46.745409 2574 patch_prober.go:28] interesting pod/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 08 04:19:46.745838 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:46.745471 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" podUID="fc67f27e-765a-425a-9b20-d7d05f43301c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 08 04:19:47.063024 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:47.062986 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:47.205022 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:47.204989 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:47.745166 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:47.745129 2574 patch_prober.go:28] interesting pod/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 08 04:19:47.745376 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:47.745210 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" podUID="fc67f27e-765a-425a-9b20-d7d05f43301c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 08 04:19:48.317028 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:48.316985 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:48.465723 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:48.465690 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:48.745603 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:48.745522 2574 patch_prober.go:28] interesting pod/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 08 04:19:48.745603 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:48.745577 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" podUID="fc67f27e-765a-425a-9b20-d7d05f43301c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 08 04:19:48.925730 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:48.925652 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:49.051932 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:49.051903 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:49.748747 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:49.748713 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:49.749115 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:49.748980 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:49.749637 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:49.749617 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:19:52.814655 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:52.814612 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:57.462703 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:57.462657 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:57.837257 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:57.837211 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:58.202416 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:58.202345 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:19:58.312679 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:19:58.312645 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:20:01.094882 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:01.094840 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:20:01.219005 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:01.218969 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:20:02.342869 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:02.342834 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:20:02.442763 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:02.442730 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:20:02.543131 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:02.543088 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:20:02.761195 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:02.761167 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:20:02.861546 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:02.861488 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:20:02.910888 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:02.910863 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh"] Sep 08 04:20:02.913924 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:02.913905 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh" Sep 08 04:20:02.916720 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:02.916700 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-720789-dockercfg-lws2v\"" Sep 08 04:20:02.916860 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:02.916844 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-720789\"" Sep 08 04:20:02.928005 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:02.927978 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh"] Sep 08 04:20:02.957399 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:02.957369 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7"] Sep 08 04:20:02.959798 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:02.959780 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7" Sep 08 04:20:02.974601 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:02.974581 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7"] Sep 08 04:20:03.033719 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:03.033633 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f150eabb-c9b7-4498-8f06-8f9243d8f058-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh\" (UID: \"f150eabb-c9b7-4498-8f06-8f9243d8f058\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh" Sep 08 04:20:03.033719 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:03.033699 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlcdr\" (UniqueName: \"kubernetes.io/projected/f150eabb-c9b7-4498-8f06-8f9243d8f058-kube-api-access-mlcdr\") pod \"payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh\" (UID: \"f150eabb-c9b7-4498-8f06-8f9243d8f058\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh" Sep 08 04:20:03.134392 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:03.134356 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f150eabb-c9b7-4498-8f06-8f9243d8f058-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh\" (UID: \"f150eabb-c9b7-4498-8f06-8f9243d8f058\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh" Sep 08 04:20:03.134612 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:03.134407 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfw2r\" (UniqueName: \"kubernetes.io/projected/83d7d5f8-f5da-406a-a976-388737723147-kube-api-access-sfw2r\") pod \"payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7\" (UID: \"83d7d5f8-f5da-406a-a976-388737723147\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7" Sep 08 04:20:03.134612 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:03.134477 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlcdr\" (UniqueName: \"kubernetes.io/projected/f150eabb-c9b7-4498-8f06-8f9243d8f058-kube-api-access-mlcdr\") pod \"payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh\" (UID: \"f150eabb-c9b7-4498-8f06-8f9243d8f058\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh" Sep 08 04:20:03.134612 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:03.134531 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83d7d5f8-f5da-406a-a976-388737723147-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7\" (UID: \"83d7d5f8-f5da-406a-a976-388737723147\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7" Sep 08 04:20:03.135135 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:03.135114 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f150eabb-c9b7-4498-8f06-8f9243d8f058-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh\" (UID: \"f150eabb-c9b7-4498-8f06-8f9243d8f058\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh" Sep 08 04:20:03.143868 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:03.143843 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlcdr\" (UniqueName: \"kubernetes.io/projected/f150eabb-c9b7-4498-8f06-8f9243d8f058-kube-api-access-mlcdr\") pod \"payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh\" (UID: \"f150eabb-c9b7-4498-8f06-8f9243d8f058\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh" Sep 08 04:20:03.223716 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:03.223688 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh" Sep 08 04:20:03.235912 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:03.235888 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfw2r\" (UniqueName: \"kubernetes.io/projected/83d7d5f8-f5da-406a-a976-388737723147-kube-api-access-sfw2r\") pod \"payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7\" (UID: \"83d7d5f8-f5da-406a-a976-388737723147\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7" Sep 08 04:20:03.236016 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:03.235941 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83d7d5f8-f5da-406a-a976-388737723147-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7\" (UID: \"83d7d5f8-f5da-406a-a976-388737723147\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7" Sep 08 04:20:03.236632 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:03.236611 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83d7d5f8-f5da-406a-a976-388737723147-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7\" (UID: \"83d7d5f8-f5da-406a-a976-388737723147\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7" Sep 08 04:20:03.246739 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:03.245921 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfw2r\" (UniqueName: \"kubernetes.io/projected/83d7d5f8-f5da-406a-a976-388737723147-kube-api-access-sfw2r\") pod \"payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7\" (UID: \"83d7d5f8-f5da-406a-a976-388737723147\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7" Sep 08 04:20:03.268848 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:03.268824 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7" Sep 08 04:20:03.507956 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:03.507923 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh"] Sep 08 04:20:03.559500 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:20:03.559473 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d7d5f8_f5da_406a_a976_388737723147.slice/crio-02737c3f7ffe307e4be8550a868f011e9128a42e402d0cfda43a5ddfef77ed23 WatchSource:0}: Error finding container 02737c3f7ffe307e4be8550a868f011e9128a42e402d0cfda43a5ddfef77ed23: Status 404 returned error can't find the container with id 02737c3f7ffe307e4be8550a868f011e9128a42e402d0cfda43a5ddfef77ed23 Sep 08 04:20:03.559719 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:03.559694 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7"] Sep 08 04:20:03.617153 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:03.617123 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh" event={"ID":"f150eabb-c9b7-4498-8f06-8f9243d8f058","Type":"ContainerStarted","Data":"96da1c398575c819a35899074950b853b021d28972a35d4199d131fca13ee7cb"} Sep 08 04:20:03.617289 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:03.617182 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh" event={"ID":"f150eabb-c9b7-4498-8f06-8f9243d8f058","Type":"ContainerStarted","Data":"6b4431c39b1b023459684751b27fb39fdf33f3c5305ea33115ccf08806d70eec"} Sep 08 04:20:03.617397 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:03.617309 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh" Sep 08 04:20:03.618605 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:03.618574 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7" event={"ID":"83d7d5f8-f5da-406a-a976-388737723147","Type":"ContainerStarted","Data":"02737c3f7ffe307e4be8550a868f011e9128a42e402d0cfda43a5ddfef77ed23"} Sep 08 04:20:03.638578 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:03.638524 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh" podStartSLOduration=1.63848967 podStartE2EDuration="1.63848967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:20:03.634809811 +0000 UTC m=+1177.557270577" watchObservedRunningTime="2026-09-08 04:20:03.63848967 +0000 UTC m=+1177.560950410" Sep 08 04:20:04.626351 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:04.626313 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7" event={"ID":"83d7d5f8-f5da-406a-a976-388737723147","Type":"ContainerStarted","Data":"6444adb957bbf186fb5e7d6f36dfaf8dfad0d8e8c252c00f156cc6a80d4f8cb6"} Sep 08 04:20:04.649419 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:04.649363 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7" podStartSLOduration=2.649343227 podStartE2EDuration="2.649343227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:20:04.647319246 +0000 UTC m=+1178.569780001" watchObservedRunningTime="2026-09-08 04:20:04.649343227 +0000 UTC m=+1178.571803972" Sep 08 04:20:05.632498 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:05.632468 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7" Sep 08 04:20:09.729754 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:09.729716 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:20:12.040192 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:12.040148 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:20:14.629051 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:14.629017 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh" Sep 08 04:20:16.646919 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:16.646871 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7" Sep 08 04:20:24.530086 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:24.530049 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:20:26.041374 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:26.041343 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:20:26.287766 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:26.287736 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:20:26.651344 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:26.651303 2574 scope.go:117] "RemoveContainer" containerID="c3673fd3ccfd0ba4708a95eb23019e4c0ddf493ba6e0f509c5854b18c237b843" Sep 08 04:20:27.124317 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:27.124285 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:20:27.534885 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:27.534844 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:20:27.675518 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:27.675463 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:20:30.364141 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:30.364094 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:20:30.471326 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:30.471301 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:20:31.603022 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:31.602981 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:20:31.701697 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:31.701659 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:20:31.849798 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:31.849765 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:20:32.213877 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:32.213834 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:20:32.326899 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:32.326871 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:20:32.352001 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:32.351972 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5"] Sep 08 04:20:32.359152 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:32.356853 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5" Sep 08 04:20:32.360229 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:32.360200 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-777fbd7c-dockercfg-rdb6q\"" Sep 08 04:20:32.360341 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:32.360265 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-777fbd7c\"" Sep 08 04:20:32.367733 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:32.367711 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5"] Sep 08 04:20:32.475207 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:32.475115 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w2hc\" (UniqueName: \"kubernetes.io/projected/667b389b-7e78-489c-82d4-851f237fbe98-kube-api-access-7w2hc\") pod \"payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5\" (UID: \"667b389b-7e78-489c-82d4-851f237fbe98\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5" Sep 08 04:20:32.475365 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:32.475223 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/667b389b-7e78-489c-82d4-851f237fbe98-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5\" (UID: \"667b389b-7e78-489c-82d4-851f237fbe98\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5" Sep 08 04:20:32.576080 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:32.576052 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w2hc\" (UniqueName: \"kubernetes.io/projected/667b389b-7e78-489c-82d4-851f237fbe98-kube-api-access-7w2hc\") pod \"payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5\" (UID: \"667b389b-7e78-489c-82d4-851f237fbe98\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5" Sep 08 04:20:32.576212 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:32.576095 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/667b389b-7e78-489c-82d4-851f237fbe98-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5\" (UID: \"667b389b-7e78-489c-82d4-851f237fbe98\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5" Sep 08 04:20:32.576791 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:32.576774 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/667b389b-7e78-489c-82d4-851f237fbe98-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5\" (UID: \"667b389b-7e78-489c-82d4-851f237fbe98\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5" Sep 08 04:20:32.586770 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:32.586742 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w2hc\" (UniqueName: \"kubernetes.io/projected/667b389b-7e78-489c-82d4-851f237fbe98-kube-api-access-7w2hc\") pod \"payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5\" (UID: \"667b389b-7e78-489c-82d4-851f237fbe98\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5" Sep 08 04:20:32.670323 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:32.670300 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5" Sep 08 04:20:32.947551 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:32.947529 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5"] Sep 08 04:20:32.949269 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:20:32.949244 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667b389b_7e78_489c_82d4_851f237fbe98.slice/crio-aaa94ca570952f97daf6c5f5e16475e165da0fd385f7bdf33f2ff6976418a83f WatchSource:0}: Error finding container aaa94ca570952f97daf6c5f5e16475e165da0fd385f7bdf33f2ff6976418a83f: Status 404 returned error can't find the container with id aaa94ca570952f97daf6c5f5e16475e165da0fd385f7bdf33f2ff6976418a83f Sep 08 04:20:33.806151 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:33.806119 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5" event={"ID":"667b389b-7e78-489c-82d4-851f237fbe98","Type":"ContainerStarted","Data":"aa6e3e08c573fb1b9ba87f56ac210c0f42f6796cbfcda4a63c3c957a624d1864"} Sep 08 04:20:33.806629 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:33.806179 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5" event={"ID":"667b389b-7e78-489c-82d4-851f237fbe98","Type":"ContainerStarted","Data":"aaa94ca570952f97daf6c5f5e16475e165da0fd385f7bdf33f2ff6976418a83f"} Sep 08 04:20:33.806629 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:33.806299 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5" Sep 08 04:20:33.831114 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:33.831072 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5" podStartSLOduration=1.831060458 podStartE2EDuration="1.831060458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:20:33.828258088 +0000 UTC m=+1207.750718851" watchObservedRunningTime="2026-09-08 04:20:33.831060458 +0000 UTC m=+1207.753521197" Sep 08 04:20:44.814393 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:44.814355 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5" Sep 08 04:20:53.359341 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:53.359310 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:20:53.435122 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:53.435093 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:20:55.333370 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:20:55.333323 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:21:04.675329 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:04.675292 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:21:09.980807 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:09.980761 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:21:25.761529 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:25.761402 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:21:25.825744 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:25.825690 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:21:26.672381 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:26.672348 2574 scope.go:117] "RemoveContainer" containerID="bf88e0a8cb10a4699793ee389e8897617deee39cb2767c77c9a90a90ebc4c55f" Sep 08 04:21:26.681374 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:26.681354 2574 scope.go:117] "RemoveContainer" containerID="9f1c4940536c429df81d9391e00724fc26280035c5c811571d9e4bfcee98cb38" Sep 08 04:21:26.689648 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:26.689628 2574 scope.go:117] "RemoveContainer" containerID="48b54313e852d54be3d45d027a1ecc79cc6bd4d15eb3eace483cf7ec3a0edd65" Sep 08 04:21:51.946033 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:51.945994 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:21:58.151746 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.151708 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk"] Sep 08 04:21:58.154866 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.154839 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" Sep 08 04:21:58.157551 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.157521 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-777fbd7c\"/\"default-dockercfg-zj5k9\"" Sep 08 04:21:58.157688 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.157558 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-777fbd7c\"/\"kube-root-ca.crt\"" Sep 08 04:21:58.158479 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.158452 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-777fbd7c\"/\"openshift-service-ca.crt\"" Sep 08 04:21:58.158833 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.158812 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-777fbd7c\"/\"e2e-lifecycle-model-777fbd7c-kserve-self-signed-certs\"" Sep 08 04:21:58.167070 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.167006 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk"] Sep 08 04:21:58.177607 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.177575 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:21:58.260464 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.260437 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8q9n\" (UniqueName: \"kubernetes.io/projected/317e28b2-e741-45ad-a207-1fad78e183e1-kube-api-access-g8q9n\") pod \"e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk\" (UID: \"317e28b2-e741-45ad-a207-1fad78e183e1\") " pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" Sep 08 04:21:58.260618 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.260473 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/317e28b2-e741-45ad-a207-1fad78e183e1-tmp-dir\") pod \"e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk\" (UID: \"317e28b2-e741-45ad-a207-1fad78e183e1\") " pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" Sep 08 04:21:58.260618 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.260566 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/317e28b2-e741-45ad-a207-1fad78e183e1-tls-certs\") pod \"e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk\" (UID: \"317e28b2-e741-45ad-a207-1fad78e183e1\") " pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" Sep 08 04:21:58.260743 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.260627 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/317e28b2-e741-45ad-a207-1fad78e183e1-home\") pod \"e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk\" (UID: \"317e28b2-e741-45ad-a207-1fad78e183e1\") " pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" Sep 08 04:21:58.260743 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.260719 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/317e28b2-e741-45ad-a207-1fad78e183e1-dshm\") pod \"e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk\" (UID: \"317e28b2-e741-45ad-a207-1fad78e183e1\") " pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" Sep 08 04:21:58.260852 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.260753 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/317e28b2-e741-45ad-a207-1fad78e183e1-model-cache\") pod \"e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk\" (UID: \"317e28b2-e741-45ad-a207-1fad78e183e1\") " pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" Sep 08 04:21:58.361812 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.361779 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/317e28b2-e741-45ad-a207-1fad78e183e1-tls-certs\") pod \"e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk\" (UID: \"317e28b2-e741-45ad-a207-1fad78e183e1\") " pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" Sep 08 04:21:58.361947 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.361818 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/317e28b2-e741-45ad-a207-1fad78e183e1-home\") pod \"e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk\" (UID: \"317e28b2-e741-45ad-a207-1fad78e183e1\") " pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" Sep 08 04:21:58.361947 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.361887 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/317e28b2-e741-45ad-a207-1fad78e183e1-dshm\") pod \"e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk\" (UID: \"317e28b2-e741-45ad-a207-1fad78e183e1\") " pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" Sep 08 04:21:58.362032 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.362014 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/317e28b2-e741-45ad-a207-1fad78e183e1-model-cache\") pod \"e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk\" (UID: \"317e28b2-e741-45ad-a207-1fad78e183e1\") " pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" Sep 08 04:21:58.362076 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.362061 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8q9n\" (UniqueName: \"kubernetes.io/projected/317e28b2-e741-45ad-a207-1fad78e183e1-kube-api-access-g8q9n\") pod \"e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk\" (UID: \"317e28b2-e741-45ad-a207-1fad78e183e1\") " pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" Sep 08 04:21:58.362123 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.362094 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/317e28b2-e741-45ad-a207-1fad78e183e1-tmp-dir\") pod \"e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk\" (UID: \"317e28b2-e741-45ad-a207-1fad78e183e1\") " pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" Sep 08 04:21:58.362288 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.362263 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/317e28b2-e741-45ad-a207-1fad78e183e1-home\") pod \"e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk\" (UID: \"317e28b2-e741-45ad-a207-1fad78e183e1\") " pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" Sep 08 04:21:58.362384 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.362363 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/317e28b2-e741-45ad-a207-1fad78e183e1-model-cache\") pod \"e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk\" (UID: \"317e28b2-e741-45ad-a207-1fad78e183e1\") " pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" Sep 08 04:21:58.362440 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.362423 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/317e28b2-e741-45ad-a207-1fad78e183e1-tmp-dir\") pod \"e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk\" (UID: \"317e28b2-e741-45ad-a207-1fad78e183e1\") " pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" Sep 08 04:21:58.364204 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.364185 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/317e28b2-e741-45ad-a207-1fad78e183e1-dshm\") pod \"e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk\" (UID: \"317e28b2-e741-45ad-a207-1fad78e183e1\") " pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" Sep 08 04:21:58.364628 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.364609 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/317e28b2-e741-45ad-a207-1fad78e183e1-tls-certs\") pod \"e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk\" (UID: \"317e28b2-e741-45ad-a207-1fad78e183e1\") " pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" Sep 08 04:21:58.371927 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.371900 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8q9n\" (UniqueName: \"kubernetes.io/projected/317e28b2-e741-45ad-a207-1fad78e183e1-kube-api-access-g8q9n\") pod \"e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk\" (UID: \"317e28b2-e741-45ad-a207-1fad78e183e1\") " pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" Sep 08 04:21:58.466761 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.466708 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" Sep 08 04:21:58.748317 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:58.748288 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk"] Sep 08 04:21:58.748741 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:21:58.748713 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod317e28b2_e741_45ad_a207_1fad78e183e1.slice/crio-177f4f69af6284c5137bdd6b24f59885f95133768df3962b4001e9992d6de165 WatchSource:0}: Error finding container 177f4f69af6284c5137bdd6b24f59885f95133768df3962b4001e9992d6de165: Status 404 returned error can't find the container with id 177f4f69af6284c5137bdd6b24f59885f95133768df3962b4001e9992d6de165 Sep 08 04:21:59.332835 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:59.332797 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" event={"ID":"317e28b2-e741-45ad-a207-1fad78e183e1","Type":"ContainerStarted","Data":"92be0974b24f19f893b2f5b97d8557ffab6bd81fe9eb978316fdb517eeab7ca7"} Sep 08 04:21:59.332835 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:59.332839 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" event={"ID":"317e28b2-e741-45ad-a207-1fad78e183e1","Type":"ContainerStarted","Data":"177f4f69af6284c5137bdd6b24f59885f95133768df3962b4001e9992d6de165"} Sep 08 04:21:59.353131 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:21:59.353081 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" podStartSLOduration=1.353064372 podStartE2EDuration="1.353064372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:21:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:21:59.350853919 +0000 UTC m=+1293.273314660" watchObservedRunningTime="2026-09-08 04:21:59.353064372 +0000 UTC m=+1293.275525112" Sep 08 04:22:05.011003 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:05.010957 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:22:05.111992 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:05.111951 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:22:08.467493 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:08.467448 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" Sep 08 04:22:08.467933 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:08.467531 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" Sep 08 04:22:08.483646 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:08.483610 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" Sep 08 04:22:09.406521 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:09.406470 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" Sep 08 04:22:13.372940 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:13.372905 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:22:19.743609 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:19.743572 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:22:25.894641 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:25.894593 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-777fbd7c-5l62z"] Sep 08 04:22:25.897831 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:25.897806 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-777fbd7c-5l62z" Sep 08 04:22:25.900602 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:25.900574 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 04:22:25.900731 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:25.900667 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7r29w\"" Sep 08 04:22:25.906327 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:25.906298 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-777fbd7c-5l62z"] Sep 08 04:22:25.988599 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:25.988572 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abd7527b-eadd-4c23-b5e9-b8ca09c2776d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-777fbd7c-5l62z\" (UID: \"abd7527b-eadd-4c23-b5e9-b8ca09c2776d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-777fbd7c-5l62z" Sep 08 04:22:26.001496 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:26.001471 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:22:26.089426 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:26.089395 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abd7527b-eadd-4c23-b5e9-b8ca09c2776d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-777fbd7c-5l62z\" (UID: \"abd7527b-eadd-4c23-b5e9-b8ca09c2776d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-777fbd7c-5l62z" Sep 08 04:22:26.090026 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:26.090008 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abd7527b-eadd-4c23-b5e9-b8ca09c2776d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-777fbd7c-5l62z\" (UID: \"abd7527b-eadd-4c23-b5e9-b8ca09c2776d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-777fbd7c-5l62z" Sep 08 04:22:26.211205 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:26.211138 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-777fbd7c-5l62z" Sep 08 04:22:26.487036 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:26.487012 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-777fbd7c-5l62z"] Sep 08 04:22:26.489638 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:22:26.489613 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd7527b_eadd_4c23_b5e9_b8ca09c2776d.slice/crio-7ce4cbe122a31423ce42e86fb0873e593444fe90955bd84b26fbda0886b8f47b WatchSource:0}: Error finding container 7ce4cbe122a31423ce42e86fb0873e593444fe90955bd84b26fbda0886b8f47b: Status 404 returned error can't find the container with id 7ce4cbe122a31423ce42e86fb0873e593444fe90955bd84b26fbda0886b8f47b Sep 08 04:22:26.500339 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:26.500311 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-777fbd7c-5l62z" event={"ID":"abd7527b-eadd-4c23-b5e9-b8ca09c2776d","Type":"ContainerStarted","Data":"7ce4cbe122a31423ce42e86fb0873e593444fe90955bd84b26fbda0886b8f47b"} Sep 08 04:22:26.720412 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:26.720389 2574 scope.go:117] "RemoveContainer" containerID="c79aa2f6484dade242cb4f03389b42bed85abca655d90b4e5c3aef3a3e1c49d5" Sep 08 04:22:26.728329 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:26.728312 2574 scope.go:117] "RemoveContainer" containerID="7a781917418de65508da2eb7a2e2cff2dc1713b0d1db5b2b88988e22e1c407c8" Sep 08 04:22:26.735793 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:26.735778 2574 scope.go:117] "RemoveContainer" containerID="4a464b67a054d9b9de29ac6df3c0a96d0363078189d696a2237317990ef14e7b" Sep 08 04:22:27.508086 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:27.508046 2574 generic.go:356] "Generic (PLEG): container finished" podID="abd7527b-eadd-4c23-b5e9-b8ca09c2776d" containerID="627bd03271bdac1a47096fbb3b7c8d0c73d2b7885193331ba5112bd9a6ef6897" exitCode=0 Sep 08 04:22:27.508433 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:27.508131 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-777fbd7c-5l62z" event={"ID":"abd7527b-eadd-4c23-b5e9-b8ca09c2776d","Type":"ContainerDied","Data":"627bd03271bdac1a47096fbb3b7c8d0c73d2b7885193331ba5112bd9a6ef6897"} Sep 08 04:22:28.643352 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:28.643330 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-777fbd7c-5l62z" Sep 08 04:22:28.713680 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:28.713657 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abd7527b-eadd-4c23-b5e9-b8ca09c2776d-maas-api-service-ca\") pod \"abd7527b-eadd-4c23-b5e9-b8ca09c2776d\" (UID: \"abd7527b-eadd-4c23-b5e9-b8ca09c2776d\") " Sep 08 04:22:28.713969 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:28.713948 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd7527b-eadd-4c23-b5e9-b8ca09c2776d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "abd7527b-eadd-4c23-b5e9-b8ca09c2776d" (UID: "abd7527b-eadd-4c23-b5e9-b8ca09c2776d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:22:28.814448 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:28.814394 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abd7527b-eadd-4c23-b5e9-b8ca09c2776d-maas-api-service-ca\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:22:29.521579 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:29.521542 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-777fbd7c-5l62z" event={"ID":"abd7527b-eadd-4c23-b5e9-b8ca09c2776d","Type":"ContainerDied","Data":"7ce4cbe122a31423ce42e86fb0873e593444fe90955bd84b26fbda0886b8f47b"} Sep 08 04:22:29.521780 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:29.521593 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ce4cbe122a31423ce42e86fb0873e593444fe90955bd84b26fbda0886b8f47b" Sep 08 04:22:29.521780 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:29.521552 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-777fbd7c-5l62z" Sep 08 04:22:41.432973 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:41.432935 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:22:41.462143 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:41.462107 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5"] Sep 08 04:22:41.462817 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:41.462781 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5" podUID="667b389b-7e78-489c-82d4-851f237fbe98" containerName="payload-pre-processing" containerID="cri-o://aa6e3e08c573fb1b9ba87f56ac210c0f42f6796cbfcda4a63c3c957a624d1864" gracePeriod=30 Sep 08 04:22:41.533274 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:41.533221 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:22:41.590770 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:41.590740 2574 generic.go:356] "Generic (PLEG): container finished" podID="667b389b-7e78-489c-82d4-851f237fbe98" containerID="aa6e3e08c573fb1b9ba87f56ac210c0f42f6796cbfcda4a63c3c957a624d1864" exitCode=0 Sep 08 04:22:41.590908 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:41.590805 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5" event={"ID":"667b389b-7e78-489c-82d4-851f237fbe98","Type":"ContainerDied","Data":"aa6e3e08c573fb1b9ba87f56ac210c0f42f6796cbfcda4a63c3c957a624d1864"} Sep 08 04:22:41.633731 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:41.633709 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:22:41.647710 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:41.647681 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-777fbd7c-5l62z"] Sep 08 04:22:41.651614 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:41.651590 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-777fbd7c-5l62z"] Sep 08 04:22:41.699828 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:41.699799 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:22:41.708734 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:41.708716 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5" Sep 08 04:22:41.818624 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:41.818595 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7w2hc\" (UniqueName: \"kubernetes.io/projected/667b389b-7e78-489c-82d4-851f237fbe98-kube-api-access-7w2hc\") pod \"667b389b-7e78-489c-82d4-851f237fbe98\" (UID: \"667b389b-7e78-489c-82d4-851f237fbe98\") " Sep 08 04:22:41.818792 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:41.818745 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/667b389b-7e78-489c-82d4-851f237fbe98-plugins-config-volume\") pod \"667b389b-7e78-489c-82d4-851f237fbe98\" (UID: \"667b389b-7e78-489c-82d4-851f237fbe98\") " Sep 08 04:22:41.819122 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:41.819091 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/667b389b-7e78-489c-82d4-851f237fbe98-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "667b389b-7e78-489c-82d4-851f237fbe98" (UID: "667b389b-7e78-489c-82d4-851f237fbe98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:22:41.820732 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:41.820705 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667b389b-7e78-489c-82d4-851f237fbe98-kube-api-access-7w2hc" (OuterVolumeSpecName: "kube-api-access-7w2hc") pod "667b389b-7e78-489c-82d4-851f237fbe98" (UID: "667b389b-7e78-489c-82d4-851f237fbe98"). InnerVolumeSpecName "kube-api-access-7w2hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:22:41.920005 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:41.919979 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/667b389b-7e78-489c-82d4-851f237fbe98-plugins-config-volume\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:22:41.920005 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:41.920003 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w2hc\" (UniqueName: \"kubernetes.io/projected/667b389b-7e78-489c-82d4-851f237fbe98-kube-api-access-7w2hc\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:22:42.574623 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:42.574590 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abd7527b-eadd-4c23-b5e9-b8ca09c2776d" path="/var/lib/kubelet/pods/abd7527b-eadd-4c23-b5e9-b8ca09c2776d/volumes" Sep 08 04:22:42.597596 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:42.597575 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5" Sep 08 04:22:42.597719 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:42.597598 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5" event={"ID":"667b389b-7e78-489c-82d4-851f237fbe98","Type":"ContainerDied","Data":"aaa94ca570952f97daf6c5f5e16475e165da0fd385f7bdf33f2ff6976418a83f"} Sep 08 04:22:42.597719 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:42.597665 2574 scope.go:117] "RemoveContainer" containerID="aa6e3e08c573fb1b9ba87f56ac210c0f42f6796cbfcda4a63c3c957a624d1864" Sep 08 04:22:42.615183 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:42.615162 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5"] Sep 08 04:22:42.619198 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:42.619179 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-777fbd7c-86676597cd-7xsb5"] Sep 08 04:22:44.568928 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:44.568891 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="667b389b-7e78-489c-82d4-851f237fbe98" path="/var/lib/kubelet/pods/667b389b-7e78-489c-82d4-851f237fbe98/volumes" Sep 08 04:22:44.870257 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:44.870176 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:22:49.858477 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:49.858399 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk"] Sep 08 04:22:49.954125 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:22:49.954092 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:23:04.878109 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:04.878044 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" podUID="317e28b2-e741-45ad-a207-1fad78e183e1" containerName="main" containerID="cri-o://92be0974b24f19f893b2f5b97d8557ffab6bd81fe9eb978316fdb517eeab7ca7" gracePeriod=45 Sep 08 04:23:05.125143 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.125121 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" Sep 08 04:23:05.225416 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.225352 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8q9n\" (UniqueName: \"kubernetes.io/projected/317e28b2-e741-45ad-a207-1fad78e183e1-kube-api-access-g8q9n\") pod \"317e28b2-e741-45ad-a207-1fad78e183e1\" (UID: \"317e28b2-e741-45ad-a207-1fad78e183e1\") " Sep 08 04:23:05.225554 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.225424 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/317e28b2-e741-45ad-a207-1fad78e183e1-tls-certs\") pod \"317e28b2-e741-45ad-a207-1fad78e183e1\" (UID: \"317e28b2-e741-45ad-a207-1fad78e183e1\") " Sep 08 04:23:05.225554 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.225534 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/317e28b2-e741-45ad-a207-1fad78e183e1-home\") pod \"317e28b2-e741-45ad-a207-1fad78e183e1\" (UID: \"317e28b2-e741-45ad-a207-1fad78e183e1\") " Sep 08 04:23:05.225632 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.225571 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/317e28b2-e741-45ad-a207-1fad78e183e1-dshm\") pod \"317e28b2-e741-45ad-a207-1fad78e183e1\" (UID: \"317e28b2-e741-45ad-a207-1fad78e183e1\") " Sep 08 04:23:05.225632 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.225618 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/317e28b2-e741-45ad-a207-1fad78e183e1-model-cache\") pod \"317e28b2-e741-45ad-a207-1fad78e183e1\" (UID: \"317e28b2-e741-45ad-a207-1fad78e183e1\") " Sep 08 04:23:05.225713 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.225642 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/317e28b2-e741-45ad-a207-1fad78e183e1-tmp-dir\") pod \"317e28b2-e741-45ad-a207-1fad78e183e1\" (UID: \"317e28b2-e741-45ad-a207-1fad78e183e1\") " Sep 08 04:23:05.225832 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.225809 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317e28b2-e741-45ad-a207-1fad78e183e1-home" (OuterVolumeSpecName: "home") pod "317e28b2-e741-45ad-a207-1fad78e183e1" (UID: "317e28b2-e741-45ad-a207-1fad78e183e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:23:05.225933 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.225897 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317e28b2-e741-45ad-a207-1fad78e183e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "317e28b2-e741-45ad-a207-1fad78e183e1" (UID: "317e28b2-e741-45ad-a207-1fad78e183e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:23:05.226098 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.226028 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/317e28b2-e741-45ad-a207-1fad78e183e1-home\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:23:05.226098 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.226047 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/317e28b2-e741-45ad-a207-1fad78e183e1-model-cache\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:23:05.226214 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.226106 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317e28b2-e741-45ad-a207-1fad78e183e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "317e28b2-e741-45ad-a207-1fad78e183e1" (UID: "317e28b2-e741-45ad-a207-1fad78e183e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:23:05.227591 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.227570 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/317e28b2-e741-45ad-a207-1fad78e183e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "317e28b2-e741-45ad-a207-1fad78e183e1" (UID: "317e28b2-e741-45ad-a207-1fad78e183e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 04:23:05.227870 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.227851 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/317e28b2-e741-45ad-a207-1fad78e183e1-kube-api-access-g8q9n" (OuterVolumeSpecName: "kube-api-access-g8q9n") pod "317e28b2-e741-45ad-a207-1fad78e183e1" (UID: "317e28b2-e741-45ad-a207-1fad78e183e1"). InnerVolumeSpecName "kube-api-access-g8q9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:23:05.227930 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.227878 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317e28b2-e741-45ad-a207-1fad78e183e1-dshm" (OuterVolumeSpecName: "dshm") pod "317e28b2-e741-45ad-a207-1fad78e183e1" (UID: "317e28b2-e741-45ad-a207-1fad78e183e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:23:05.327303 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.327276 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/317e28b2-e741-45ad-a207-1fad78e183e1-dshm\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:23:05.327303 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.327299 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/317e28b2-e741-45ad-a207-1fad78e183e1-tmp-dir\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:23:05.327438 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.327312 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8q9n\" (UniqueName: \"kubernetes.io/projected/317e28b2-e741-45ad-a207-1fad78e183e1-kube-api-access-g8q9n\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:23:05.327438 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.327326 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/317e28b2-e741-45ad-a207-1fad78e183e1-tls-certs\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:23:05.732784 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.732707 2574 generic.go:356] "Generic (PLEG): container finished" podID="317e28b2-e741-45ad-a207-1fad78e183e1" containerID="92be0974b24f19f893b2f5b97d8557ffab6bd81fe9eb978316fdb517eeab7ca7" exitCode=0 Sep 08 04:23:05.732784 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.732754 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" event={"ID":"317e28b2-e741-45ad-a207-1fad78e183e1","Type":"ContainerDied","Data":"92be0974b24f19f893b2f5b97d8557ffab6bd81fe9eb978316fdb517eeab7ca7"} Sep 08 04:23:05.732784 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.732787 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" event={"ID":"317e28b2-e741-45ad-a207-1fad78e183e1","Type":"ContainerDied","Data":"177f4f69af6284c5137bdd6b24f59885f95133768df3962b4001e9992d6de165"} Sep 08 04:23:05.733080 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.732784 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk" Sep 08 04:23:05.733080 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.732806 2574 scope.go:117] "RemoveContainer" containerID="92be0974b24f19f893b2f5b97d8557ffab6bd81fe9eb978316fdb517eeab7ca7" Sep 08 04:23:05.746146 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.746122 2574 scope.go:117] "RemoveContainer" containerID="92be0974b24f19f893b2f5b97d8557ffab6bd81fe9eb978316fdb517eeab7ca7" Sep 08 04:23:05.746556 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:23:05.746519 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92be0974b24f19f893b2f5b97d8557ffab6bd81fe9eb978316fdb517eeab7ca7\": container with ID starting with 92be0974b24f19f893b2f5b97d8557ffab6bd81fe9eb978316fdb517eeab7ca7 not found: ID does not exist" containerID="92be0974b24f19f893b2f5b97d8557ffab6bd81fe9eb978316fdb517eeab7ca7" Sep 08 04:23:05.746653 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.746560 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92be0974b24f19f893b2f5b97d8557ffab6bd81fe9eb978316fdb517eeab7ca7"} err="failed to get container status \"92be0974b24f19f893b2f5b97d8557ffab6bd81fe9eb978316fdb517eeab7ca7\": rpc error: code = NotFound desc = could not find container \"92be0974b24f19f893b2f5b97d8557ffab6bd81fe9eb978316fdb517eeab7ca7\": container with ID starting with 92be0974b24f19f893b2f5b97d8557ffab6bd81fe9eb978316fdb517eeab7ca7 not found: ID does not exist" Sep 08 04:23:05.758639 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.758612 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk"] Sep 08 04:23:05.762497 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:05.762475 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-777fbd7c/e2e-lifecycle-model-777fbd7c-kserve-5899775fd7-88kdk"] Sep 08 04:23:06.571382 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:06.571348 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="317e28b2-e741-45ad-a207-1fad78e183e1" path="/var/lib/kubelet/pods/317e28b2-e741-45ad-a207-1fad78e183e1/volumes" Sep 08 04:23:26.191156 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:26.191122 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:23:27.411800 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:27.411765 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:23:33.107003 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:33.106969 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:23:33.193652 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:33.193618 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:23:33.384491 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:33.384408 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:23:34.691572 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:34.691535 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:23:34.787627 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:34.787590 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:23:34.893804 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:34.893762 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:23:35.554267 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:35.554228 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:23:35.645710 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:35.645672 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:23:35.745743 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:35.745711 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:23:37.121117 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:37.121084 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:23:43.522161 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:43.522121 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:23:43.586859 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:43.586827 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:23:44.223163 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:44.223134 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:23:50.140404 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:50.140363 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:23:50.247283 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:50.247249 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:23:50.340244 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:23:50.340210 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:00.868849 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:00.868818 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:01.111384 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:01.111354 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:22.119062 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:22.118962 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:26.293262 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:26.293228 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:26.394424 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:26.394399 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:26.781038 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:26.781011 2574 scope.go:117] "RemoveContainer" containerID="e5c1dc7f49c52797c5e39e5949b5be2b0c99a9650adc3ef04731b63f7144f90d" Sep 08 04:24:28.174431 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:28.174395 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:28.272409 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:28.272382 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:28.523062 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:28.523036 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:28.880482 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:28.880406 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:29.245245 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:29.245151 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:31.727271 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:31.727236 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:31.824813 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:31.824783 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:32.883877 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:32.883847 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:32.961670 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:32.961636 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:33.327405 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.327371 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:33.496230 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.496200 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:33.591068 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.590998 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:33.630080 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.630050 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dab019-cbd64957-fdk2x"] Sep 08 04:24:33.630401 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.630389 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abd7527b-eadd-4c23-b5e9-b8ca09c2776d" containerName="revoke-keys" Sep 08 04:24:33.630446 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.630403 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="abd7527b-eadd-4c23-b5e9-b8ca09c2776d" containerName="revoke-keys" Sep 08 04:24:33.630446 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.630417 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="317e28b2-e741-45ad-a207-1fad78e183e1" containerName="main" Sep 08 04:24:33.630446 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.630423 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="317e28b2-e741-45ad-a207-1fad78e183e1" containerName="main" Sep 08 04:24:33.630446 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.630440 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="667b389b-7e78-489c-82d4-851f237fbe98" containerName="payload-pre-processing" Sep 08 04:24:33.630446 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.630447 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="667b389b-7e78-489c-82d4-851f237fbe98" containerName="payload-pre-processing" Sep 08 04:24:33.630636 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.630498 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="317e28b2-e741-45ad-a207-1fad78e183e1" containerName="main" Sep 08 04:24:33.630636 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.630522 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="abd7527b-eadd-4c23-b5e9-b8ca09c2776d" containerName="revoke-keys" Sep 08 04:24:33.630636 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.630529 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="667b389b-7e78-489c-82d4-851f237fbe98" containerName="payload-pre-processing" Sep 08 04:24:33.634891 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.634875 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dab019-cbd64957-fdk2x" Sep 08 04:24:33.638091 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.638068 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-dab019-dockercfg-8xjww\"" Sep 08 04:24:33.638439 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.638423 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-dab019\"" Sep 08 04:24:33.646394 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.646368 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dab019-cbd64957-fdk2x"] Sep 08 04:24:33.668924 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.668898 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw"] Sep 08 04:24:33.672280 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.672265 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw" Sep 08 04:24:33.685683 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.685657 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw"] Sep 08 04:24:33.760661 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.760627 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e393bdf7-d610-48df-b7cd-f823ad24b68b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dab019-cbd64957-fdk2x\" (UID: \"e393bdf7-d610-48df-b7cd-f823ad24b68b\") " pod="openshift-ingress/payload-processing-e2e-api-a-dab019-cbd64957-fdk2x" Sep 08 04:24:33.760661 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.760659 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee61684-c968-43b8-9b20-b21b56685c2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw\" (UID: \"7ee61684-c968-43b8-9b20-b21b56685c2f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw" Sep 08 04:24:33.760865 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.760784 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9ns9\" (UniqueName: \"kubernetes.io/projected/e393bdf7-d610-48df-b7cd-f823ad24b68b-kube-api-access-z9ns9\") pod \"payload-processing-e2e-api-a-dab019-cbd64957-fdk2x\" (UID: \"e393bdf7-d610-48df-b7cd-f823ad24b68b\") " pod="openshift-ingress/payload-processing-e2e-api-a-dab019-cbd64957-fdk2x" Sep 08 04:24:33.760865 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.760844 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qspg8\" (UniqueName: \"kubernetes.io/projected/7ee61684-c968-43b8-9b20-b21b56685c2f-kube-api-access-qspg8\") pod \"payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw\" (UID: \"7ee61684-c968-43b8-9b20-b21b56685c2f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw" Sep 08 04:24:33.861229 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.861156 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e393bdf7-d610-48df-b7cd-f823ad24b68b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dab019-cbd64957-fdk2x\" (UID: \"e393bdf7-d610-48df-b7cd-f823ad24b68b\") " pod="openshift-ingress/payload-processing-e2e-api-a-dab019-cbd64957-fdk2x" Sep 08 04:24:33.861229 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.861182 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee61684-c968-43b8-9b20-b21b56685c2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw\" (UID: \"7ee61684-c968-43b8-9b20-b21b56685c2f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw" Sep 08 04:24:33.861381 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.861236 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9ns9\" (UniqueName: \"kubernetes.io/projected/e393bdf7-d610-48df-b7cd-f823ad24b68b-kube-api-access-z9ns9\") pod \"payload-processing-e2e-api-a-dab019-cbd64957-fdk2x\" (UID: \"e393bdf7-d610-48df-b7cd-f823ad24b68b\") " pod="openshift-ingress/payload-processing-e2e-api-a-dab019-cbd64957-fdk2x" Sep 08 04:24:33.861381 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.861285 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qspg8\" (UniqueName: \"kubernetes.io/projected/7ee61684-c968-43b8-9b20-b21b56685c2f-kube-api-access-qspg8\") pod \"payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw\" (UID: \"7ee61684-c968-43b8-9b20-b21b56685c2f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw" Sep 08 04:24:33.861914 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.861895 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e393bdf7-d610-48df-b7cd-f823ad24b68b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dab019-cbd64957-fdk2x\" (UID: \"e393bdf7-d610-48df-b7cd-f823ad24b68b\") " pod="openshift-ingress/payload-processing-e2e-api-a-dab019-cbd64957-fdk2x" Sep 08 04:24:33.861958 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.861927 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee61684-c968-43b8-9b20-b21b56685c2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw\" (UID: \"7ee61684-c968-43b8-9b20-b21b56685c2f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw" Sep 08 04:24:33.870931 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.870906 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qspg8\" (UniqueName: \"kubernetes.io/projected/7ee61684-c968-43b8-9b20-b21b56685c2f-kube-api-access-qspg8\") pod \"payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw\" (UID: \"7ee61684-c968-43b8-9b20-b21b56685c2f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw" Sep 08 04:24:33.871017 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.870991 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9ns9\" (UniqueName: \"kubernetes.io/projected/e393bdf7-d610-48df-b7cd-f823ad24b68b-kube-api-access-z9ns9\") pod \"payload-processing-e2e-api-a-dab019-cbd64957-fdk2x\" (UID: \"e393bdf7-d610-48df-b7cd-f823ad24b68b\") " pod="openshift-ingress/payload-processing-e2e-api-a-dab019-cbd64957-fdk2x" Sep 08 04:24:33.945395 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.945375 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dab019-cbd64957-fdk2x" Sep 08 04:24:33.983847 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:33.983642 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw" Sep 08 04:24:34.218929 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:34.218903 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dab019-cbd64957-fdk2x"] Sep 08 04:24:34.219776 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:24:34.219753 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode393bdf7_d610_48df_b7cd_f823ad24b68b.slice/crio-2711f56e423d5821fe86c9e7af32dfd5307b453e2637fa21d644e37472a1cda8 WatchSource:0}: Error finding container 2711f56e423d5821fe86c9e7af32dfd5307b453e2637fa21d644e37472a1cda8: Status 404 returned error can't find the container with id 2711f56e423d5821fe86c9e7af32dfd5307b453e2637fa21d644e37472a1cda8 Sep 08 04:24:34.252891 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:34.252865 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dab019-cbd64957-fdk2x" event={"ID":"e393bdf7-d610-48df-b7cd-f823ad24b68b","Type":"ContainerStarted","Data":"2711f56e423d5821fe86c9e7af32dfd5307b453e2637fa21d644e37472a1cda8"} Sep 08 04:24:34.269584 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:34.269563 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw"] Sep 08 04:24:34.270468 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:24:34.270447 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee61684_c968_43b8_9b20_b21b56685c2f.slice/crio-8ed71d5e4c81e1aa8e4cf83d597a1add7fe7a3fa2986b7bbd15b82f05e44b1d6 WatchSource:0}: Error finding container 8ed71d5e4c81e1aa8e4cf83d597a1add7fe7a3fa2986b7bbd15b82f05e44b1d6: Status 404 returned error can't find the container with id 8ed71d5e4c81e1aa8e4cf83d597a1add7fe7a3fa2986b7bbd15b82f05e44b1d6 Sep 08 04:24:35.259577 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:35.259543 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw" event={"ID":"7ee61684-c968-43b8-9b20-b21b56685c2f","Type":"ContainerStarted","Data":"3b38f2a369f3c54bb696b76620d493a18c3dc5873b025c3d904f870232396c05"} Sep 08 04:24:35.260029 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:35.259602 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw" event={"ID":"7ee61684-c968-43b8-9b20-b21b56685c2f","Type":"ContainerStarted","Data":"8ed71d5e4c81e1aa8e4cf83d597a1add7fe7a3fa2986b7bbd15b82f05e44b1d6"} Sep 08 04:24:35.260029 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:35.259714 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw" Sep 08 04:24:35.261270 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:35.261246 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dab019-cbd64957-fdk2x" event={"ID":"e393bdf7-d610-48df-b7cd-f823ad24b68b","Type":"ContainerStarted","Data":"48c3fd063ded997085b947f954f150018401cc69de2afd65ceb239249bcb6bcb"} Sep 08 04:24:35.261395 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:35.261313 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-dab019-cbd64957-fdk2x" Sep 08 04:24:35.278517 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:35.278461 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw" podStartSLOduration=2.278449218 podStartE2EDuration="2.278449218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:24:35.276471144 +0000 UTC m=+1449.198931899" watchObservedRunningTime="2026-09-08 04:24:35.278449218 +0000 UTC m=+1449.200910018" Sep 08 04:24:35.296040 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:35.295998 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-dab019-cbd64957-fdk2x" podStartSLOduration=2.295980912 podStartE2EDuration="2.295980912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:24:35.29272522 +0000 UTC m=+1449.215185982" watchObservedRunningTime="2026-09-08 04:24:35.295980912 +0000 UTC m=+1449.218441655" Sep 08 04:24:35.968642 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:35.968596 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:37.170546 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:37.170493 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:43.172045 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:43.172007 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:46.269144 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:46.269111 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-dab019-cbd64957-fdk2x" Sep 08 04:24:46.269582 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:46.269194 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw" Sep 08 04:24:53.927557 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:53.927474 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:54.865146 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:54.865112 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:55.679864 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:55.679823 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:55.972763 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:55.972673 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:56.758083 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:56.758053 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:57.210892 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:57.210807 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:57.576159 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:57.576129 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:24:57.841780 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:24:57.841682 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:25:00.941169 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:00.941134 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:25:01.039276 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:01.039245 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:25:02.101463 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:02.101424 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:25:02.177778 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:02.177745 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:25:02.250812 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:02.250779 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:25:02.673081 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:02.673052 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:25:02.777483 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:02.777444 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:25:02.974202 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:02.974118 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s"] Sep 08 04:25:02.978034 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:02.978012 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s" Sep 08 04:25:02.983894 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:02.983871 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-7bac07\"" Sep 08 04:25:02.984341 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:02.984321 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-7bac07-dockercfg-rkkh4\"" Sep 08 04:25:02.990151 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:02.990126 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s"] Sep 08 04:25:03.004627 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:03.004603 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp4bj\" (UniqueName: \"kubernetes.io/projected/89f60991-2227-48b2-87b9-0c0c135bb799-kube-api-access-gp4bj\") pod \"payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s\" (UID: \"89f60991-2227-48b2-87b9-0c0c135bb799\") " pod="openshift-ingress/payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s" Sep 08 04:25:03.004777 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:03.004752 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89f60991-2227-48b2-87b9-0c0c135bb799-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s\" (UID: \"89f60991-2227-48b2-87b9-0c0c135bb799\") " pod="openshift-ingress/payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s" Sep 08 04:25:03.105449 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:03.105422 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp4bj\" (UniqueName: \"kubernetes.io/projected/89f60991-2227-48b2-87b9-0c0c135bb799-kube-api-access-gp4bj\") pod \"payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s\" (UID: \"89f60991-2227-48b2-87b9-0c0c135bb799\") " pod="openshift-ingress/payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s" Sep 08 04:25:03.105850 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:03.105484 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89f60991-2227-48b2-87b9-0c0c135bb799-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s\" (UID: \"89f60991-2227-48b2-87b9-0c0c135bb799\") " pod="openshift-ingress/payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s" Sep 08 04:25:03.106241 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:03.106217 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89f60991-2227-48b2-87b9-0c0c135bb799-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s\" (UID: \"89f60991-2227-48b2-87b9-0c0c135bb799\") " pod="openshift-ingress/payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s" Sep 08 04:25:03.119354 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:03.119334 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp4bj\" (UniqueName: \"kubernetes.io/projected/89f60991-2227-48b2-87b9-0c0c135bb799-kube-api-access-gp4bj\") pod \"payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s\" (UID: \"89f60991-2227-48b2-87b9-0c0c135bb799\") " pod="openshift-ingress/payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s" Sep 08 04:25:03.289000 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:03.288978 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s" Sep 08 04:25:03.563790 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:03.563768 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s"] Sep 08 04:25:03.566078 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:25:03.566049 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89f60991_2227_48b2_87b9_0c0c135bb799.slice/crio-c07a5bf423315af75a3afab214505d432275af8aaf08ba023a4614dbdb9afb2d WatchSource:0}: Error finding container c07a5bf423315af75a3afab214505d432275af8aaf08ba023a4614dbdb9afb2d: Status 404 returned error can't find the container with id c07a5bf423315af75a3afab214505d432275af8aaf08ba023a4614dbdb9afb2d Sep 08 04:25:04.433816 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:04.433787 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s" event={"ID":"89f60991-2227-48b2-87b9-0c0c135bb799","Type":"ContainerStarted","Data":"b7d54d493cc71fac19f328f56e2affdd356a18252b5a605965ad8c2c428c40ec"} Sep 08 04:25:04.434195 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:04.433842 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s" event={"ID":"89f60991-2227-48b2-87b9-0c0c135bb799","Type":"ContainerStarted","Data":"c07a5bf423315af75a3afab214505d432275af8aaf08ba023a4614dbdb9afb2d"} Sep 08 04:25:04.434195 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:04.433869 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s" Sep 08 04:25:04.450679 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:04.450626 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s" podStartSLOduration=2.450607546 podStartE2EDuration="2.450607546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:25:04.45015418 +0000 UTC m=+1478.372614946" watchObservedRunningTime="2026-09-08 04:25:04.450607546 +0000 UTC m=+1478.373068287" Sep 08 04:25:10.649830 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:10.649786 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:25:15.442320 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:15.442287 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s" Sep 08 04:25:24.084725 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:24.084678 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:25:25.617652 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:25.617601 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:25:25.991417 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:25.991333 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:25:26.801623 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:26.801575 2574 scope.go:117] "RemoveContainer" containerID="55071f61f5b9e527f79f3605bbc6d38f462416a6ad93cc592f5ce0f8a34a6c67" Sep 08 04:25:26.810162 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:26.810143 2574 scope.go:117] "RemoveContainer" containerID="1c04b92941c729acd03937c1565991c39f9e9e78f794a26df8707e253434df1d" Sep 08 04:25:26.817951 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:26.817931 2574 scope.go:117] "RemoveContainer" containerID="ca551dc1061c8965158e55752054f732b2826688183fd61bb4a33e60978b0b47" Sep 08 04:25:29.032162 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:29.032129 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dab019-57j2d"] Sep 08 04:25:29.035709 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:29.035692 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dab019-57j2d" Sep 08 04:25:29.038151 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:29.038132 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 04:25:29.038265 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:29.038225 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7r29w\"" Sep 08 04:25:29.043839 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:29.043819 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dab019-57j2d"] Sep 08 04:25:29.135362 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:29.135336 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff64036-0181-4d5b-bf57-8eb41b169fc1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dab019-57j2d\" (UID: \"7ff64036-0181-4d5b-bf57-8eb41b169fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dab019-57j2d" Sep 08 04:25:29.180780 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:29.180751 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:25:29.236215 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:29.236192 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff64036-0181-4d5b-bf57-8eb41b169fc1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dab019-57j2d\" (UID: \"7ff64036-0181-4d5b-bf57-8eb41b169fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dab019-57j2d" Sep 08 04:25:29.237614 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:29.237153 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff64036-0181-4d5b-bf57-8eb41b169fc1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dab019-57j2d\" (UID: \"7ff64036-0181-4d5b-bf57-8eb41b169fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dab019-57j2d" Sep 08 04:25:29.347609 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:29.347497 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dab019-57j2d" Sep 08 04:25:29.618052 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:29.618022 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dab019-57j2d"] Sep 08 04:25:29.618959 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:25:29.618930 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff64036_0181_4d5b_bf57_8eb41b169fc1.slice/crio-ad185d32444b90efa5af6d6e545e8bf77cd45c025ce358f0c2b0f1925c594055 WatchSource:0}: Error finding container ad185d32444b90efa5af6d6e545e8bf77cd45c025ce358f0c2b0f1925c594055: Status 404 returned error can't find the container with id ad185d32444b90efa5af6d6e545e8bf77cd45c025ce358f0c2b0f1925c594055 Sep 08 04:25:30.592208 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:30.592138 2574 generic.go:356] "Generic (PLEG): container finished" podID="7ff64036-0181-4d5b-bf57-8eb41b169fc1" containerID="6f48382327c762d648a059d299970d96a78ccfc084b88425f5bda621c09ff63c" exitCode=0 Sep 08 04:25:30.592562 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:30.592238 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dab019-57j2d" event={"ID":"7ff64036-0181-4d5b-bf57-8eb41b169fc1","Type":"ContainerDied","Data":"6f48382327c762d648a059d299970d96a78ccfc084b88425f5bda621c09ff63c"} Sep 08 04:25:30.592562 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:30.592301 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dab019-57j2d" event={"ID":"7ff64036-0181-4d5b-bf57-8eb41b169fc1","Type":"ContainerStarted","Data":"ad185d32444b90efa5af6d6e545e8bf77cd45c025ce358f0c2b0f1925c594055"} Sep 08 04:25:31.732389 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:31.732367 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dab019-57j2d" Sep 08 04:25:31.858804 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:31.858738 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff64036-0181-4d5b-bf57-8eb41b169fc1-maas-api-service-ca\") pod \"7ff64036-0181-4d5b-bf57-8eb41b169fc1\" (UID: \"7ff64036-0181-4d5b-bf57-8eb41b169fc1\") " Sep 08 04:25:31.859062 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:31.859039 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ff64036-0181-4d5b-bf57-8eb41b169fc1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7ff64036-0181-4d5b-bf57-8eb41b169fc1" (UID: "7ff64036-0181-4d5b-bf57-8eb41b169fc1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:25:31.959442 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:31.959413 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff64036-0181-4d5b-bf57-8eb41b169fc1-maas-api-service-ca\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:25:32.603643 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:32.603619 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dab019-57j2d" Sep 08 04:25:32.603643 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:32.603630 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dab019-57j2d" event={"ID":"7ff64036-0181-4d5b-bf57-8eb41b169fc1","Type":"ContainerDied","Data":"ad185d32444b90efa5af6d6e545e8bf77cd45c025ce358f0c2b0f1925c594055"} Sep 08 04:25:32.603791 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:32.603673 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad185d32444b90efa5af6d6e545e8bf77cd45c025ce358f0c2b0f1925c594055" Sep 08 04:25:37.486545 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:37.486445 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:25:44.562404 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:44.562358 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:25:44.589744 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:44.589715 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dab019-cbd64957-fdk2x"] Sep 08 04:25:44.589945 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:44.589920 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-dab019-cbd64957-fdk2x" podUID="e393bdf7-d610-48df-b7cd-f823ad24b68b" containerName="payload-processing" containerID="cri-o://48c3fd063ded997085b947f954f150018401cc69de2afd65ceb239249bcb6bcb" gracePeriod=30 Sep 08 04:25:44.595119 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:44.595077 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw"] Sep 08 04:25:44.595376 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:44.595347 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw" podUID="7ee61684-c968-43b8-9b20-b21b56685c2f" containerName="payload-pre-processing" containerID="cri-o://3b38f2a369f3c54bb696b76620d493a18c3dc5873b025c3d904f870232396c05" gracePeriod=30 Sep 08 04:25:44.665590 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:44.665541 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:25:44.763410 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:44.763381 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:25:44.779147 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:44.779119 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dab019-57j2d"] Sep 08 04:25:44.783120 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:44.783096 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dab019-57j2d"] Sep 08 04:25:44.867713 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:44.867688 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:25:44.870384 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:44.870367 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dab019-cbd64957-fdk2x" Sep 08 04:25:44.873457 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:44.873440 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw" Sep 08 04:25:44.966171 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:44.966132 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e393bdf7-d610-48df-b7cd-f823ad24b68b-plugins-config-volume\") pod \"e393bdf7-d610-48df-b7cd-f823ad24b68b\" (UID: \"e393bdf7-d610-48df-b7cd-f823ad24b68b\") " Sep 08 04:25:44.966385 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:44.966218 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9ns9\" (UniqueName: \"kubernetes.io/projected/e393bdf7-d610-48df-b7cd-f823ad24b68b-kube-api-access-z9ns9\") pod \"e393bdf7-d610-48df-b7cd-f823ad24b68b\" (UID: \"e393bdf7-d610-48df-b7cd-f823ad24b68b\") " Sep 08 04:25:44.966385 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:44.966248 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee61684-c968-43b8-9b20-b21b56685c2f-plugins-config-volume\") pod \"7ee61684-c968-43b8-9b20-b21b56685c2f\" (UID: \"7ee61684-c968-43b8-9b20-b21b56685c2f\") " Sep 08 04:25:44.966569 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:44.966390 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qspg8\" (UniqueName: \"kubernetes.io/projected/7ee61684-c968-43b8-9b20-b21b56685c2f-kube-api-access-qspg8\") pod \"7ee61684-c968-43b8-9b20-b21b56685c2f\" (UID: \"7ee61684-c968-43b8-9b20-b21b56685c2f\") " Sep 08 04:25:44.966966 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:44.966935 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee61684-c968-43b8-9b20-b21b56685c2f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7ee61684-c968-43b8-9b20-b21b56685c2f" (UID: "7ee61684-c968-43b8-9b20-b21b56685c2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:25:44.966966 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:44.966945 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e393bdf7-d610-48df-b7cd-f823ad24b68b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e393bdf7-d610-48df-b7cd-f823ad24b68b" (UID: "e393bdf7-d610-48df-b7cd-f823ad24b68b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:25:44.969365 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:44.969333 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee61684-c968-43b8-9b20-b21b56685c2f-kube-api-access-qspg8" (OuterVolumeSpecName: "kube-api-access-qspg8") pod "7ee61684-c968-43b8-9b20-b21b56685c2f" (UID: "7ee61684-c968-43b8-9b20-b21b56685c2f"). InnerVolumeSpecName "kube-api-access-qspg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:25:44.969468 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:44.969407 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e393bdf7-d610-48df-b7cd-f823ad24b68b-kube-api-access-z9ns9" (OuterVolumeSpecName: "kube-api-access-z9ns9") pod "e393bdf7-d610-48df-b7cd-f823ad24b68b" (UID: "e393bdf7-d610-48df-b7cd-f823ad24b68b"). InnerVolumeSpecName "kube-api-access-z9ns9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:25:45.067688 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:45.067666 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e393bdf7-d610-48df-b7cd-f823ad24b68b-plugins-config-volume\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:25:45.067688 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:45.067688 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9ns9\" (UniqueName: \"kubernetes.io/projected/e393bdf7-d610-48df-b7cd-f823ad24b68b-kube-api-access-z9ns9\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:25:45.067807 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:45.067699 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee61684-c968-43b8-9b20-b21b56685c2f-plugins-config-volume\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:25:45.067807 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:45.067708 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qspg8\" (UniqueName: \"kubernetes.io/projected/7ee61684-c968-43b8-9b20-b21b56685c2f-kube-api-access-qspg8\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:25:45.158703 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:45.158643 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:25:45.688176 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:45.688150 2574 generic.go:356] "Generic (PLEG): container finished" podID="7ee61684-c968-43b8-9b20-b21b56685c2f" containerID="3b38f2a369f3c54bb696b76620d493a18c3dc5873b025c3d904f870232396c05" exitCode=0 Sep 08 04:25:45.688539 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:45.688199 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw" Sep 08 04:25:45.688539 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:45.688240 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw" event={"ID":"7ee61684-c968-43b8-9b20-b21b56685c2f","Type":"ContainerDied","Data":"3b38f2a369f3c54bb696b76620d493a18c3dc5873b025c3d904f870232396c05"} Sep 08 04:25:45.688539 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:45.688304 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw" event={"ID":"7ee61684-c968-43b8-9b20-b21b56685c2f","Type":"ContainerDied","Data":"8ed71d5e4c81e1aa8e4cf83d597a1add7fe7a3fa2986b7bbd15b82f05e44b1d6"} Sep 08 04:25:45.688539 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:45.688339 2574 scope.go:117] "RemoveContainer" containerID="3b38f2a369f3c54bb696b76620d493a18c3dc5873b025c3d904f870232396c05" Sep 08 04:25:45.689898 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:45.689875 2574 generic.go:356] "Generic (PLEG): container finished" podID="e393bdf7-d610-48df-b7cd-f823ad24b68b" containerID="48c3fd063ded997085b947f954f150018401cc69de2afd65ceb239249bcb6bcb" exitCode=0 Sep 08 04:25:45.690008 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:45.689918 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dab019-cbd64957-fdk2x" Sep 08 04:25:45.690008 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:45.689969 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dab019-cbd64957-fdk2x" event={"ID":"e393bdf7-d610-48df-b7cd-f823ad24b68b","Type":"ContainerDied","Data":"48c3fd063ded997085b947f954f150018401cc69de2afd65ceb239249bcb6bcb"} Sep 08 04:25:45.690086 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:45.690023 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dab019-cbd64957-fdk2x" event={"ID":"e393bdf7-d610-48df-b7cd-f823ad24b68b","Type":"ContainerDied","Data":"2711f56e423d5821fe86c9e7af32dfd5307b453e2637fa21d644e37472a1cda8"} Sep 08 04:25:45.699652 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:45.699628 2574 scope.go:117] "RemoveContainer" containerID="3b38f2a369f3c54bb696b76620d493a18c3dc5873b025c3d904f870232396c05" Sep 08 04:25:45.699991 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:25:45.699963 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b38f2a369f3c54bb696b76620d493a18c3dc5873b025c3d904f870232396c05\": container with ID starting with 3b38f2a369f3c54bb696b76620d493a18c3dc5873b025c3d904f870232396c05 not found: ID does not exist" containerID="3b38f2a369f3c54bb696b76620d493a18c3dc5873b025c3d904f870232396c05" Sep 08 04:25:45.700102 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:45.700003 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b38f2a369f3c54bb696b76620d493a18c3dc5873b025c3d904f870232396c05"} err="failed to get container status \"3b38f2a369f3c54bb696b76620d493a18c3dc5873b025c3d904f870232396c05\": rpc error: code = NotFound desc = could not find container \"3b38f2a369f3c54bb696b76620d493a18c3dc5873b025c3d904f870232396c05\": container with ID starting with 3b38f2a369f3c54bb696b76620d493a18c3dc5873b025c3d904f870232396c05 not found: ID does not exist" Sep 08 04:25:45.700102 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:45.700027 2574 scope.go:117] "RemoveContainer" containerID="48c3fd063ded997085b947f954f150018401cc69de2afd65ceb239249bcb6bcb" Sep 08 04:25:45.708643 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:45.708623 2574 scope.go:117] "RemoveContainer" containerID="48c3fd063ded997085b947f954f150018401cc69de2afd65ceb239249bcb6bcb" Sep 08 04:25:45.708914 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:25:45.708895 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48c3fd063ded997085b947f954f150018401cc69de2afd65ceb239249bcb6bcb\": container with ID starting with 48c3fd063ded997085b947f954f150018401cc69de2afd65ceb239249bcb6bcb not found: ID does not exist" containerID="48c3fd063ded997085b947f954f150018401cc69de2afd65ceb239249bcb6bcb" Sep 08 04:25:45.708959 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:45.708921 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48c3fd063ded997085b947f954f150018401cc69de2afd65ceb239249bcb6bcb"} err="failed to get container status \"48c3fd063ded997085b947f954f150018401cc69de2afd65ceb239249bcb6bcb\": rpc error: code = NotFound desc = could not find container \"48c3fd063ded997085b947f954f150018401cc69de2afd65ceb239249bcb6bcb\": container with ID starting with 48c3fd063ded997085b947f954f150018401cc69de2afd65ceb239249bcb6bcb not found: ID does not exist" Sep 08 04:25:45.718450 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:45.718424 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw"] Sep 08 04:25:45.722355 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:45.722332 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dab019-7d88b56774-xklfw"] Sep 08 04:25:45.736924 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:45.736905 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dab019-cbd64957-fdk2x"] Sep 08 04:25:45.739333 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:45.739308 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dab019-cbd64957-fdk2x"] Sep 08 04:25:46.568417 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:46.568380 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ee61684-c968-43b8-9b20-b21b56685c2f" path="/var/lib/kubelet/pods/7ee61684-c968-43b8-9b20-b21b56685c2f/volumes" Sep 08 04:25:46.568760 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:46.568744 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ff64036-0181-4d5b-bf57-8eb41b169fc1" path="/var/lib/kubelet/pods/7ff64036-0181-4d5b-bf57-8eb41b169fc1/volumes" Sep 08 04:25:46.569069 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:46.569057 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e393bdf7-d610-48df-b7cd-f823ad24b68b" path="/var/lib/kubelet/pods/e393bdf7-d610-48df-b7cd-f823ad24b68b/volumes" Sep 08 04:25:51.448496 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:51.448414 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:25:51.572366 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:51.572332 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:25:51.778483 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:51.778449 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:25:52.159890 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:52.159812 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bac07-h7jwr"] Sep 08 04:25:52.160233 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:52.160219 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e393bdf7-d610-48df-b7cd-f823ad24b68b" containerName="payload-processing" Sep 08 04:25:52.160277 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:52.160234 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e393bdf7-d610-48df-b7cd-f823ad24b68b" containerName="payload-processing" Sep 08 04:25:52.160277 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:52.160244 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee61684-c968-43b8-9b20-b21b56685c2f" containerName="payload-pre-processing" Sep 08 04:25:52.160277 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:52.160249 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee61684-c968-43b8-9b20-b21b56685c2f" containerName="payload-pre-processing" Sep 08 04:25:52.160277 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:52.160262 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ff64036-0181-4d5b-bf57-8eb41b169fc1" containerName="revoke-keys" Sep 08 04:25:52.160277 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:52.160267 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ff64036-0181-4d5b-bf57-8eb41b169fc1" containerName="revoke-keys" Sep 08 04:25:52.160433 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:52.160326 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee61684-c968-43b8-9b20-b21b56685c2f" containerName="payload-pre-processing" Sep 08 04:25:52.160433 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:52.160333 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e393bdf7-d610-48df-b7cd-f823ad24b68b" containerName="payload-processing" Sep 08 04:25:52.160433 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:52.160343 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ff64036-0181-4d5b-bf57-8eb41b169fc1" containerName="revoke-keys" Sep 08 04:25:52.164875 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:52.164858 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bac07-h7jwr" Sep 08 04:25:52.167753 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:52.167726 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7r29w\"" Sep 08 04:25:52.167955 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:52.167941 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 04:25:52.173678 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:52.173658 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bac07-h7jwr"] Sep 08 04:25:52.325418 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:52.325393 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02810032-3b80-4026-917c-c1bdb60e28e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7bac07-h7jwr\" (UID: \"02810032-3b80-4026-917c-c1bdb60e28e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bac07-h7jwr" Sep 08 04:25:52.406852 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:52.406824 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:25:52.426107 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:52.426058 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02810032-3b80-4026-917c-c1bdb60e28e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7bac07-h7jwr\" (UID: \"02810032-3b80-4026-917c-c1bdb60e28e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bac07-h7jwr" Sep 08 04:25:52.426648 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:52.426630 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02810032-3b80-4026-917c-c1bdb60e28e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7bac07-h7jwr\" (UID: \"02810032-3b80-4026-917c-c1bdb60e28e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bac07-h7jwr" Sep 08 04:25:52.476232 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:52.476202 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bac07-h7jwr" Sep 08 04:25:52.760761 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:52.760735 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bac07-h7jwr"] Sep 08 04:25:52.762328 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:25:52.762294 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02810032_3b80_4026_917c_c1bdb60e28e9.slice/crio-19ea0a0d22eeeedce1d43ecd35af6da245d973f23780660a5b19ed92ade6ac43 WatchSource:0}: Error finding container 19ea0a0d22eeeedce1d43ecd35af6da245d973f23780660a5b19ed92ade6ac43: Status 404 returned error can't find the container with id 19ea0a0d22eeeedce1d43ecd35af6da245d973f23780660a5b19ed92ade6ac43 Sep 08 04:25:53.741603 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:53.741575 2574 generic.go:356] "Generic (PLEG): container finished" podID="02810032-3b80-4026-917c-c1bdb60e28e9" containerID="7e085045221a7591bd16f75481cb302f7829eed22ee218945b4f810a8a8e13d9" exitCode=0 Sep 08 04:25:53.742003 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:53.741664 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bac07-h7jwr" event={"ID":"02810032-3b80-4026-917c-c1bdb60e28e9","Type":"ContainerDied","Data":"7e085045221a7591bd16f75481cb302f7829eed22ee218945b4f810a8a8e13d9"} Sep 08 04:25:53.742003 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:53.741724 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bac07-h7jwr" event={"ID":"02810032-3b80-4026-917c-c1bdb60e28e9","Type":"ContainerStarted","Data":"19ea0a0d22eeeedce1d43ecd35af6da245d973f23780660a5b19ed92ade6ac43"} Sep 08 04:25:54.884622 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:54.884600 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bac07-h7jwr" Sep 08 04:25:55.051397 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:55.051324 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02810032-3b80-4026-917c-c1bdb60e28e9-maas-api-service-ca\") pod \"02810032-3b80-4026-917c-c1bdb60e28e9\" (UID: \"02810032-3b80-4026-917c-c1bdb60e28e9\") " Sep 08 04:25:55.051696 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:55.051669 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02810032-3b80-4026-917c-c1bdb60e28e9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "02810032-3b80-4026-917c-c1bdb60e28e9" (UID: "02810032-3b80-4026-917c-c1bdb60e28e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:25:55.152043 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:55.152019 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02810032-3b80-4026-917c-c1bdb60e28e9-maas-api-service-ca\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:25:55.754904 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:55.754877 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bac07-h7jwr" Sep 08 04:25:55.754904 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:55.754893 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bac07-h7jwr" event={"ID":"02810032-3b80-4026-917c-c1bdb60e28e9","Type":"ContainerDied","Data":"19ea0a0d22eeeedce1d43ecd35af6da245d973f23780660a5b19ed92ade6ac43"} Sep 08 04:25:55.755055 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:25:55.754938 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19ea0a0d22eeeedce1d43ecd35af6da245d973f23780660a5b19ed92ade6ac43" Sep 08 04:26:07.949664 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:07.949627 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:26:07.985589 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:07.985552 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s"] Sep 08 04:26:07.985831 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:07.985783 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s" podUID="89f60991-2227-48b2-87b9-0c0c135bb799" containerName="payload-processing" containerID="cri-o://b7d54d493cc71fac19f328f56e2affdd356a18252b5a605965ad8c2c428c40ec" gracePeriod=30 Sep 08 04:26:08.179077 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:08.179023 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bac07-h7jwr"] Sep 08 04:26:08.184084 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:08.184056 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7bac07-h7jwr"] Sep 08 04:26:08.238176 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:08.238157 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s" Sep 08 04:26:08.354193 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:08.354161 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gp4bj\" (UniqueName: \"kubernetes.io/projected/89f60991-2227-48b2-87b9-0c0c135bb799-kube-api-access-gp4bj\") pod \"89f60991-2227-48b2-87b9-0c0c135bb799\" (UID: \"89f60991-2227-48b2-87b9-0c0c135bb799\") " Sep 08 04:26:08.354390 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:08.354213 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89f60991-2227-48b2-87b9-0c0c135bb799-plugins-config-volume\") pod \"89f60991-2227-48b2-87b9-0c0c135bb799\" (UID: \"89f60991-2227-48b2-87b9-0c0c135bb799\") " Sep 08 04:26:08.354675 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:08.354646 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89f60991-2227-48b2-87b9-0c0c135bb799-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "89f60991-2227-48b2-87b9-0c0c135bb799" (UID: "89f60991-2227-48b2-87b9-0c0c135bb799"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:26:08.356309 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:08.356286 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f60991-2227-48b2-87b9-0c0c135bb799-kube-api-access-gp4bj" (OuterVolumeSpecName: "kube-api-access-gp4bj") pod "89f60991-2227-48b2-87b9-0c0c135bb799" (UID: "89f60991-2227-48b2-87b9-0c0c135bb799"). InnerVolumeSpecName "kube-api-access-gp4bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:26:08.455853 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:08.455829 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp4bj\" (UniqueName: \"kubernetes.io/projected/89f60991-2227-48b2-87b9-0c0c135bb799-kube-api-access-gp4bj\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:26:08.455853 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:08.455851 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89f60991-2227-48b2-87b9-0c0c135bb799-plugins-config-volume\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:26:08.568113 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:08.568083 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02810032-3b80-4026-917c-c1bdb60e28e9" path="/var/lib/kubelet/pods/02810032-3b80-4026-917c-c1bdb60e28e9/volumes" Sep 08 04:26:08.828947 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:08.828889 2574 generic.go:356] "Generic (PLEG): container finished" podID="89f60991-2227-48b2-87b9-0c0c135bb799" containerID="b7d54d493cc71fac19f328f56e2affdd356a18252b5a605965ad8c2c428c40ec" exitCode=0 Sep 08 04:26:08.828947 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:08.828935 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s" Sep 08 04:26:08.829099 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:08.828967 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s" event={"ID":"89f60991-2227-48b2-87b9-0c0c135bb799","Type":"ContainerDied","Data":"b7d54d493cc71fac19f328f56e2affdd356a18252b5a605965ad8c2c428c40ec"} Sep 08 04:26:08.829099 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:08.829012 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s" event={"ID":"89f60991-2227-48b2-87b9-0c0c135bb799","Type":"ContainerDied","Data":"c07a5bf423315af75a3afab214505d432275af8aaf08ba023a4614dbdb9afb2d"} Sep 08 04:26:08.829099 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:08.829034 2574 scope.go:117] "RemoveContainer" containerID="b7d54d493cc71fac19f328f56e2affdd356a18252b5a605965ad8c2c428c40ec" Sep 08 04:26:08.837982 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:08.837967 2574 scope.go:117] "RemoveContainer" containerID="b7d54d493cc71fac19f328f56e2affdd356a18252b5a605965ad8c2c428c40ec" Sep 08 04:26:08.838238 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:26:08.838223 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7d54d493cc71fac19f328f56e2affdd356a18252b5a605965ad8c2c428c40ec\": container with ID starting with b7d54d493cc71fac19f328f56e2affdd356a18252b5a605965ad8c2c428c40ec not found: ID does not exist" containerID="b7d54d493cc71fac19f328f56e2affdd356a18252b5a605965ad8c2c428c40ec" Sep 08 04:26:08.838288 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:08.838245 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7d54d493cc71fac19f328f56e2affdd356a18252b5a605965ad8c2c428c40ec"} err="failed to get container status \"b7d54d493cc71fac19f328f56e2affdd356a18252b5a605965ad8c2c428c40ec\": rpc error: code = NotFound desc = could not find container \"b7d54d493cc71fac19f328f56e2affdd356a18252b5a605965ad8c2c428c40ec\": container with ID starting with b7d54d493cc71fac19f328f56e2affdd356a18252b5a605965ad8c2c428c40ec not found: ID does not exist" Sep 08 04:26:08.848020 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:08.847999 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s"] Sep 08 04:26:08.853239 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:08.853218 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7bac07-659df5cddf-nl27s"] Sep 08 04:26:09.147547 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:09.147457 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:26:10.568278 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:10.568231 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89f60991-2227-48b2-87b9-0c0c135bb799" path="/var/lib/kubelet/pods/89f60991-2227-48b2-87b9-0c0c135bb799/volumes" Sep 08 04:26:10.753551 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:10.753525 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:26:14.431796 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:14.431737 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:26:14.512040 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:14.511978 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:26:14.717137 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:14.717043 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:26:15.049614 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:15.049578 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-75264d-qwxvq"] Sep 08 04:26:15.049975 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:15.049962 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02810032-3b80-4026-917c-c1bdb60e28e9" containerName="revoke-keys" Sep 08 04:26:15.050027 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:15.049977 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="02810032-3b80-4026-917c-c1bdb60e28e9" containerName="revoke-keys" Sep 08 04:26:15.050027 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:15.050000 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89f60991-2227-48b2-87b9-0c0c135bb799" containerName="payload-processing" Sep 08 04:26:15.050027 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:15.050006 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="89f60991-2227-48b2-87b9-0c0c135bb799" containerName="payload-processing" Sep 08 04:26:15.050139 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:15.050064 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="89f60991-2227-48b2-87b9-0c0c135bb799" containerName="payload-processing" Sep 08 04:26:15.050139 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:15.050077 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="02810032-3b80-4026-917c-c1bdb60e28e9" containerName="revoke-keys" Sep 08 04:26:15.054574 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:15.054554 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-75264d-qwxvq" Sep 08 04:26:15.057539 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:15.057491 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 04:26:15.057674 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:15.057654 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7r29w\"" Sep 08 04:26:15.059778 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:15.059753 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-75264d-qwxvq"] Sep 08 04:26:15.112534 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:15.112491 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1803bfb3-698d-41f3-8b65-fa037c30b223-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-75264d-qwxvq\" (UID: \"1803bfb3-698d-41f3-8b65-fa037c30b223\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-75264d-qwxvq" Sep 08 04:26:15.154595 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:15.154574 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:26:15.214236 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:15.213620 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1803bfb3-698d-41f3-8b65-fa037c30b223-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-75264d-qwxvq\" (UID: \"1803bfb3-698d-41f3-8b65-fa037c30b223\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-75264d-qwxvq" Sep 08 04:26:15.214761 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:15.214738 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1803bfb3-698d-41f3-8b65-fa037c30b223-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-75264d-qwxvq\" (UID: \"1803bfb3-698d-41f3-8b65-fa037c30b223\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-75264d-qwxvq" Sep 08 04:26:15.367086 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:15.367017 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-75264d-qwxvq" Sep 08 04:26:15.639365 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:15.639335 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-75264d-qwxvq"] Sep 08 04:26:15.641055 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:26:15.641031 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1803bfb3_698d_41f3_8b65_fa037c30b223.slice/crio-40ea02f04a8fbb53eb4b0abdf61eaad87d5993f00aa39fe0934853b9f559f8b7 WatchSource:0}: Error finding container 40ea02f04a8fbb53eb4b0abdf61eaad87d5993f00aa39fe0934853b9f559f8b7: Status 404 returned error can't find the container with id 40ea02f04a8fbb53eb4b0abdf61eaad87d5993f00aa39fe0934853b9f559f8b7 Sep 08 04:26:15.871665 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:15.871637 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-75264d-qwxvq" event={"ID":"1803bfb3-698d-41f3-8b65-fa037c30b223","Type":"ContainerStarted","Data":"40ea02f04a8fbb53eb4b0abdf61eaad87d5993f00aa39fe0934853b9f559f8b7"} Sep 08 04:26:16.879947 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:16.879916 2574 generic.go:356] "Generic (PLEG): container finished" podID="1803bfb3-698d-41f3-8b65-fa037c30b223" containerID="0813f790b56152b1735561f58c070d6aa8304c6daa2d56e8ef29fa2906cd46bf" exitCode=0 Sep 08 04:26:16.880310 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:16.879980 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-75264d-qwxvq" event={"ID":"1803bfb3-698d-41f3-8b65-fa037c30b223","Type":"ContainerDied","Data":"0813f790b56152b1735561f58c070d6aa8304c6daa2d56e8ef29fa2906cd46bf"} Sep 08 04:26:18.020795 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:18.020771 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-75264d-qwxvq" Sep 08 04:26:18.141546 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:18.141486 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1803bfb3-698d-41f3-8b65-fa037c30b223-maas-api-service-ca\") pod \"1803bfb3-698d-41f3-8b65-fa037c30b223\" (UID: \"1803bfb3-698d-41f3-8b65-fa037c30b223\") " Sep 08 04:26:18.141782 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:18.141760 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1803bfb3-698d-41f3-8b65-fa037c30b223-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1803bfb3-698d-41f3-8b65-fa037c30b223" (UID: "1803bfb3-698d-41f3-8b65-fa037c30b223"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:26:18.242390 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:18.242365 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1803bfb3-698d-41f3-8b65-fa037c30b223-maas-api-service-ca\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:26:18.896640 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:18.896571 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-75264d-qwxvq" Sep 08 04:26:18.896640 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:18.896580 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-75264d-qwxvq" event={"ID":"1803bfb3-698d-41f3-8b65-fa037c30b223","Type":"ContainerDied","Data":"40ea02f04a8fbb53eb4b0abdf61eaad87d5993f00aa39fe0934853b9f559f8b7"} Sep 08 04:26:18.896801 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:18.896644 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40ea02f04a8fbb53eb4b0abdf61eaad87d5993f00aa39fe0934853b9f559f8b7" Sep 08 04:26:23.049374 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:23.049340 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:26:23.990870 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:23.990837 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:26:25.660357 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:25.660323 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:26:25.723592 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:25.723560 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw"] Sep 08 04:26:25.723834 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:25.723808 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw" podUID="39d4d068-2ae5-4796-9399-8a2cb889d7e7" containerName="payload-processing" containerID="cri-o://9dac01312ba5657e083cfaae106140525f9c1c707efc3934530d569b988b6e6b" gracePeriod=30 Sep 08 04:26:25.736947 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:25.736921 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz"] Sep 08 04:26:25.737151 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:25.737130 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz" podUID="07080ec2-37dc-425a-b8e4-d4843da3250d" containerName="payload-pre-processing" containerID="cri-o://bccae318b15a48c05d893c125900bd45e153fd8324992ea8ff8f36d5d20dc5b5" gracePeriod=30 Sep 08 04:26:25.773471 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:25.773445 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:26:25.916513 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:25.916413 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:26:25.942562 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:25.942326 2574 generic.go:356] "Generic (PLEG): container finished" podID="07080ec2-37dc-425a-b8e4-d4843da3250d" containerID="bccae318b15a48c05d893c125900bd45e153fd8324992ea8ff8f36d5d20dc5b5" exitCode=0 Sep 08 04:26:25.942562 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:25.942483 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz" event={"ID":"07080ec2-37dc-425a-b8e4-d4843da3250d","Type":"ContainerDied","Data":"bccae318b15a48c05d893c125900bd45e153fd8324992ea8ff8f36d5d20dc5b5"} Sep 08 04:26:25.945201 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:25.945026 2574 generic.go:356] "Generic (PLEG): container finished" podID="39d4d068-2ae5-4796-9399-8a2cb889d7e7" containerID="9dac01312ba5657e083cfaae106140525f9c1c707efc3934530d569b988b6e6b" exitCode=0 Sep 08 04:26:25.945201 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:25.945142 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw" event={"ID":"39d4d068-2ae5-4796-9399-8a2cb889d7e7","Type":"ContainerDied","Data":"9dac01312ba5657e083cfaae106140525f9c1c707efc3934530d569b988b6e6b"} Sep 08 04:26:25.995485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:25.995461 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw" Sep 08 04:26:26.023872 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:26.023847 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz" Sep 08 04:26:26.032263 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:26.032236 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-75264d-qwxvq"] Sep 08 04:26:26.035603 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:26.035581 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-75264d-qwxvq"] Sep 08 04:26:26.044821 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:26.044797 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:26:26.104794 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:26.104773 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39d4d068-2ae5-4796-9399-8a2cb889d7e7-plugins-config-volume\") pod \"39d4d068-2ae5-4796-9399-8a2cb889d7e7\" (UID: \"39d4d068-2ae5-4796-9399-8a2cb889d7e7\") " Sep 08 04:26:26.104897 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:26.104807 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kgr6\" (UniqueName: \"kubernetes.io/projected/39d4d068-2ae5-4796-9399-8a2cb889d7e7-kube-api-access-6kgr6\") pod \"39d4d068-2ae5-4796-9399-8a2cb889d7e7\" (UID: \"39d4d068-2ae5-4796-9399-8a2cb889d7e7\") " Sep 08 04:26:26.104897 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:26.104825 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07080ec2-37dc-425a-b8e4-d4843da3250d-plugins-config-volume\") pod \"07080ec2-37dc-425a-b8e4-d4843da3250d\" (UID: \"07080ec2-37dc-425a-b8e4-d4843da3250d\") " Sep 08 04:26:26.104897 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:26.104860 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2f2q\" (UniqueName: \"kubernetes.io/projected/07080ec2-37dc-425a-b8e4-d4843da3250d-kube-api-access-j2f2q\") pod \"07080ec2-37dc-425a-b8e4-d4843da3250d\" (UID: \"07080ec2-37dc-425a-b8e4-d4843da3250d\") " Sep 08 04:26:26.105170 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:26.105150 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07080ec2-37dc-425a-b8e4-d4843da3250d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "07080ec2-37dc-425a-b8e4-d4843da3250d" (UID: "07080ec2-37dc-425a-b8e4-d4843da3250d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:26:26.105237 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:26.105173 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39d4d068-2ae5-4796-9399-8a2cb889d7e7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "39d4d068-2ae5-4796-9399-8a2cb889d7e7" (UID: "39d4d068-2ae5-4796-9399-8a2cb889d7e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:26:26.106975 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:26.106955 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07080ec2-37dc-425a-b8e4-d4843da3250d-kube-api-access-j2f2q" (OuterVolumeSpecName: "kube-api-access-j2f2q") pod "07080ec2-37dc-425a-b8e4-d4843da3250d" (UID: "07080ec2-37dc-425a-b8e4-d4843da3250d"). InnerVolumeSpecName "kube-api-access-j2f2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:26:26.107046 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:26.106988 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d4d068-2ae5-4796-9399-8a2cb889d7e7-kube-api-access-6kgr6" (OuterVolumeSpecName: "kube-api-access-6kgr6") pod "39d4d068-2ae5-4796-9399-8a2cb889d7e7" (UID: "39d4d068-2ae5-4796-9399-8a2cb889d7e7"). InnerVolumeSpecName "kube-api-access-6kgr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:26:26.206474 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:26.206419 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39d4d068-2ae5-4796-9399-8a2cb889d7e7-plugins-config-volume\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:26:26.206474 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:26.206441 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kgr6\" (UniqueName: \"kubernetes.io/projected/39d4d068-2ae5-4796-9399-8a2cb889d7e7-kube-api-access-6kgr6\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:26:26.206474 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:26.206451 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07080ec2-37dc-425a-b8e4-d4843da3250d-plugins-config-volume\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:26:26.206474 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:26.206460 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2f2q\" (UniqueName: \"kubernetes.io/projected/07080ec2-37dc-425a-b8e4-d4843da3250d-kube-api-access-j2f2q\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:26:26.388713 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:26.388681 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:26:26.567930 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:26.567895 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1803bfb3-698d-41f3-8b65-fa037c30b223" path="/var/lib/kubelet/pods/1803bfb3-698d-41f3-8b65-fa037c30b223/volumes" Sep 08 04:26:26.845317 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:26.845259 2574 scope.go:117] "RemoveContainer" containerID="bccae318b15a48c05d893c125900bd45e153fd8324992ea8ff8f36d5d20dc5b5" Sep 08 04:26:26.853901 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:26.853880 2574 scope.go:117] "RemoveContainer" containerID="9dac01312ba5657e083cfaae106140525f9c1c707efc3934530d569b988b6e6b" Sep 08 04:26:26.952604 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:26.951329 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw" Sep 08 04:26:26.952604 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:26.951921 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw" event={"ID":"39d4d068-2ae5-4796-9399-8a2cb889d7e7","Type":"ContainerDied","Data":"c658ac88c15d1e7347bcb769a9fc98125c44d4d1ccd587ad107427bf7d2db363"} Sep 08 04:26:26.952604 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:26.952011 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz" event={"ID":"07080ec2-37dc-425a-b8e4-d4843da3250d","Type":"ContainerDied","Data":"8ae46eb572c7e5930a78a8b8184a39e465f34d715b2f5a842e297b64b6a97486"} Sep 08 04:26:26.952604 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:26.952096 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz" Sep 08 04:26:27.001010 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:27.000987 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw"] Sep 08 04:26:27.006903 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:27.006857 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-75264d-6dff9b45-rhzbw"] Sep 08 04:26:27.016855 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:27.016836 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz"] Sep 08 04:26:27.023702 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:27.023684 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-75264d-b5468674b-62dwz"] Sep 08 04:26:28.568805 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:28.568769 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07080ec2-37dc-425a-b8e4-d4843da3250d" path="/var/lib/kubelet/pods/07080ec2-37dc-425a-b8e4-d4843da3250d/volumes" Sep 08 04:26:28.569196 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:28.569136 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39d4d068-2ae5-4796-9399-8a2cb889d7e7" path="/var/lib/kubelet/pods/39d4d068-2ae5-4796-9399-8a2cb889d7e7/volumes" Sep 08 04:26:33.095812 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:33.095776 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:26:33.217167 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:33.217138 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:26:33.347639 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:33.347555 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf"] Sep 08 04:26:33.347853 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:33.347824 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" podUID="078876db-1081-4a8a-bbc4-278589a1f9ad" containerName="istio-proxy" containerID="cri-o://61af56d59574e2335f02e6aa4c6075554c59ff6ca4bed1c6f82b6786dfef23b0" gracePeriod=30 Sep 08 04:26:33.384121 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:33.384092 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:26:33.713728 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:33.713648 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c2eafd-7t9x2"] Sep 08 04:26:33.714037 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:33.714023 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39d4d068-2ae5-4796-9399-8a2cb889d7e7" containerName="payload-processing" Sep 08 04:26:33.714085 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:33.714038 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="39d4d068-2ae5-4796-9399-8a2cb889d7e7" containerName="payload-processing" Sep 08 04:26:33.714085 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:33.714050 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1803bfb3-698d-41f3-8b65-fa037c30b223" containerName="revoke-keys" Sep 08 04:26:33.714085 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:33.714056 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1803bfb3-698d-41f3-8b65-fa037c30b223" containerName="revoke-keys" Sep 08 04:26:33.714085 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:33.714064 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07080ec2-37dc-425a-b8e4-d4843da3250d" containerName="payload-pre-processing" Sep 08 04:26:33.714085 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:33.714070 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="07080ec2-37dc-425a-b8e4-d4843da3250d" containerName="payload-pre-processing" Sep 08 04:26:33.714257 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:33.714125 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="39d4d068-2ae5-4796-9399-8a2cb889d7e7" containerName="payload-processing" Sep 08 04:26:33.714257 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:33.714132 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="07080ec2-37dc-425a-b8e4-d4843da3250d" containerName="payload-pre-processing" Sep 08 04:26:33.714257 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:33.714141 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1803bfb3-698d-41f3-8b65-fa037c30b223" containerName="revoke-keys" Sep 08 04:26:33.717978 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:33.717962 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c2eafd-7t9x2" Sep 08 04:26:33.722335 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:33.722317 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7r29w\"" Sep 08 04:26:33.722658 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:33.722643 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 04:26:33.728530 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:33.728493 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c2eafd-7t9x2"] Sep 08 04:26:33.857167 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:33.857142 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:26:33.870426 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:33.870399 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc94268c-d21a-49dc-a876-1a33cc55cfff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c2eafd-7t9x2\" (UID: \"dc94268c-d21a-49dc-a876-1a33cc55cfff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c2eafd-7t9x2" Sep 08 04:26:33.971744 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:33.971672 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc94268c-d21a-49dc-a876-1a33cc55cfff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c2eafd-7t9x2\" (UID: \"dc94268c-d21a-49dc-a876-1a33cc55cfff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c2eafd-7t9x2" Sep 08 04:26:33.972256 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:33.972239 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc94268c-d21a-49dc-a876-1a33cc55cfff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c2eafd-7t9x2\" (UID: \"dc94268c-d21a-49dc-a876-1a33cc55cfff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c2eafd-7t9x2" Sep 08 04:26:34.028596 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:34.028573 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c2eafd-7t9x2" Sep 08 04:26:34.311641 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:34.311614 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c2eafd-7t9x2"] Sep 08 04:26:34.312568 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:26:34.312539 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc94268c_d21a_49dc_a876_1a33cc55cfff.slice/crio-823dac0b4469cc9a597fce231806b234dfeea584d15f7be9821bc1ca2d002eb0 WatchSource:0}: Error finding container 823dac0b4469cc9a597fce231806b234dfeea584d15f7be9821bc1ca2d002eb0: Status 404 returned error can't find the container with id 823dac0b4469cc9a597fce231806b234dfeea584d15f7be9821bc1ca2d002eb0 Sep 08 04:26:35.001461 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:35.001422 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c2eafd-7t9x2" event={"ID":"dc94268c-d21a-49dc-a876-1a33cc55cfff","Type":"ContainerStarted","Data":"021b51a92aeabd30ab1a087130a972de28f13260a66bd66510bf0c5735329eef"} Sep 08 04:26:35.001697 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:35.001493 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c2eafd-7t9x2" event={"ID":"dc94268c-d21a-49dc-a876-1a33cc55cfff","Type":"ContainerStarted","Data":"823dac0b4469cc9a597fce231806b234dfeea584d15f7be9821bc1ca2d002eb0"} Sep 08 04:26:36.007839 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:36.007800 2574 generic.go:356] "Generic (PLEG): container finished" podID="dc94268c-d21a-49dc-a876-1a33cc55cfff" containerID="021b51a92aeabd30ab1a087130a972de28f13260a66bd66510bf0c5735329eef" exitCode=0 Sep 08 04:26:36.008217 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:36.007876 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c2eafd-7t9x2" event={"ID":"dc94268c-d21a-49dc-a876-1a33cc55cfff","Type":"ContainerDied","Data":"021b51a92aeabd30ab1a087130a972de28f13260a66bd66510bf0c5735329eef"} Sep 08 04:26:36.146295 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:36.146272 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c2eafd-7t9x2" Sep 08 04:26:36.290218 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:36.290157 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc94268c-d21a-49dc-a876-1a33cc55cfff-maas-api-service-ca\") pod \"dc94268c-d21a-49dc-a876-1a33cc55cfff\" (UID: \"dc94268c-d21a-49dc-a876-1a33cc55cfff\") " Sep 08 04:26:36.290498 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:36.290478 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc94268c-d21a-49dc-a876-1a33cc55cfff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dc94268c-d21a-49dc-a876-1a33cc55cfff" (UID: "dc94268c-d21a-49dc-a876-1a33cc55cfff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:26:36.390936 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:36.390912 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc94268c-d21a-49dc-a876-1a33cc55cfff-maas-api-service-ca\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:26:37.014269 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:37.014236 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c2eafd-7t9x2" event={"ID":"dc94268c-d21a-49dc-a876-1a33cc55cfff","Type":"ContainerDied","Data":"823dac0b4469cc9a597fce231806b234dfeea584d15f7be9821bc1ca2d002eb0"} Sep 08 04:26:37.014652 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:37.014284 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="823dac0b4469cc9a597fce231806b234dfeea584d15f7be9821bc1ca2d002eb0" Sep 08 04:26:37.014652 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:37.014253 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c2eafd-7t9x2" Sep 08 04:26:38.589687 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.589664 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:26:38.711216 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.711133 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/078876db-1081-4a8a-bbc4-278589a1f9ad-istiod-ca-cert\") pod \"078876db-1081-4a8a-bbc4-278589a1f9ad\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " Sep 08 04:26:38.711380 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.711220 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/078876db-1081-4a8a-bbc4-278589a1f9ad-istio-token\") pod \"078876db-1081-4a8a-bbc4-278589a1f9ad\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " Sep 08 04:26:38.711380 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.711242 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-istio-envoy\") pod \"078876db-1081-4a8a-bbc4-278589a1f9ad\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " Sep 08 04:26:38.711380 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.711258 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-istio-data\") pod \"078876db-1081-4a8a-bbc4-278589a1f9ad\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " Sep 08 04:26:38.711380 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.711273 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/078876db-1081-4a8a-bbc4-278589a1f9ad-istio-podinfo\") pod \"078876db-1081-4a8a-bbc4-278589a1f9ad\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " Sep 08 04:26:38.711380 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.711294 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-workload-socket\") pod \"078876db-1081-4a8a-bbc4-278589a1f9ad\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " Sep 08 04:26:38.711380 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.711325 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7jzn\" (UniqueName: \"kubernetes.io/projected/078876db-1081-4a8a-bbc4-278589a1f9ad-kube-api-access-s7jzn\") pod \"078876db-1081-4a8a-bbc4-278589a1f9ad\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " Sep 08 04:26:38.711732 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.711392 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-workload-certs\") pod \"078876db-1081-4a8a-bbc4-278589a1f9ad\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " Sep 08 04:26:38.711732 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.711425 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-credential-socket\") pod \"078876db-1081-4a8a-bbc4-278589a1f9ad\" (UID: \"078876db-1081-4a8a-bbc4-278589a1f9ad\") " Sep 08 04:26:38.711732 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.711595 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/078876db-1081-4a8a-bbc4-278589a1f9ad-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "078876db-1081-4a8a-bbc4-278589a1f9ad" (UID: "078876db-1081-4a8a-bbc4-278589a1f9ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:26:38.711848 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.711697 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "078876db-1081-4a8a-bbc4-278589a1f9ad" (UID: "078876db-1081-4a8a-bbc4-278589a1f9ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:26:38.711848 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.711763 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-istio-data" (OuterVolumeSpecName: "istio-data") pod "078876db-1081-4a8a-bbc4-278589a1f9ad" (UID: "078876db-1081-4a8a-bbc4-278589a1f9ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:26:38.711931 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.711880 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "078876db-1081-4a8a-bbc4-278589a1f9ad" (UID: "078876db-1081-4a8a-bbc4-278589a1f9ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:26:38.711931 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.711913 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "078876db-1081-4a8a-bbc4-278589a1f9ad" (UID: "078876db-1081-4a8a-bbc4-278589a1f9ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:26:38.713592 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.713566 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "078876db-1081-4a8a-bbc4-278589a1f9ad" (UID: "078876db-1081-4a8a-bbc4-278589a1f9ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:26:38.713755 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.713737 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078876db-1081-4a8a-bbc4-278589a1f9ad-istio-token" (OuterVolumeSpecName: "istio-token") pod "078876db-1081-4a8a-bbc4-278589a1f9ad" (UID: "078876db-1081-4a8a-bbc4-278589a1f9ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:26:38.713848 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.713772 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/078876db-1081-4a8a-bbc4-278589a1f9ad-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "078876db-1081-4a8a-bbc4-278589a1f9ad" (UID: "078876db-1081-4a8a-bbc4-278589a1f9ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 04:26:38.713848 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.713811 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078876db-1081-4a8a-bbc4-278589a1f9ad-kube-api-access-s7jzn" (OuterVolumeSpecName: "kube-api-access-s7jzn") pod "078876db-1081-4a8a-bbc4-278589a1f9ad" (UID: "078876db-1081-4a8a-bbc4-278589a1f9ad"). InnerVolumeSpecName "kube-api-access-s7jzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:26:38.812437 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.812413 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-workload-certs\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:26:38.812437 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.812435 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-credential-socket\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:26:38.812582 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.812445 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/078876db-1081-4a8a-bbc4-278589a1f9ad-istiod-ca-cert\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:26:38.812582 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.812453 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/078876db-1081-4a8a-bbc4-278589a1f9ad-istio-token\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:26:38.812582 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.812461 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-istio-envoy\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:26:38.812582 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.812468 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-istio-data\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:26:38.812582 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.812475 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/078876db-1081-4a8a-bbc4-278589a1f9ad-istio-podinfo\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:26:38.812582 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.812483 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/078876db-1081-4a8a-bbc4-278589a1f9ad-workload-socket\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:26:38.812582 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:38.812490 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7jzn\" (UniqueName: \"kubernetes.io/projected/078876db-1081-4a8a-bbc4-278589a1f9ad-kube-api-access-s7jzn\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:26:39.027295 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:39.027271 2574 generic.go:356] "Generic (PLEG): container finished" podID="078876db-1081-4a8a-bbc4-278589a1f9ad" containerID="61af56d59574e2335f02e6aa4c6075554c59ff6ca4bed1c6f82b6786dfef23b0" exitCode=0 Sep 08 04:26:39.027396 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:39.027321 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" Sep 08 04:26:39.027396 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:39.027352 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" event={"ID":"078876db-1081-4a8a-bbc4-278589a1f9ad","Type":"ContainerDied","Data":"61af56d59574e2335f02e6aa4c6075554c59ff6ca4bed1c6f82b6786dfef23b0"} Sep 08 04:26:39.027473 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:39.027405 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf" event={"ID":"078876db-1081-4a8a-bbc4-278589a1f9ad","Type":"ContainerDied","Data":"4d7b30e3eb7924d3e8b353f279a32d05ae6448a7592a151fe330cfbd6988f668"} Sep 08 04:26:39.027473 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:39.027430 2574 scope.go:117] "RemoveContainer" containerID="61af56d59574e2335f02e6aa4c6075554c59ff6ca4bed1c6f82b6786dfef23b0" Sep 08 04:26:39.036712 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:39.036697 2574 scope.go:117] "RemoveContainer" containerID="8f5126313d4618c4d9615a0c2ec95a43c1c83bf9cf176522a465efd7c925e7c2" Sep 08 04:26:39.044705 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:39.044688 2574 scope.go:117] "RemoveContainer" containerID="61af56d59574e2335f02e6aa4c6075554c59ff6ca4bed1c6f82b6786dfef23b0" Sep 08 04:26:39.044981 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:26:39.044960 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61af56d59574e2335f02e6aa4c6075554c59ff6ca4bed1c6f82b6786dfef23b0\": container with ID starting with 61af56d59574e2335f02e6aa4c6075554c59ff6ca4bed1c6f82b6786dfef23b0 not found: ID does not exist" containerID="61af56d59574e2335f02e6aa4c6075554c59ff6ca4bed1c6f82b6786dfef23b0" Sep 08 04:26:39.045071 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:39.044987 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61af56d59574e2335f02e6aa4c6075554c59ff6ca4bed1c6f82b6786dfef23b0"} err="failed to get container status \"61af56d59574e2335f02e6aa4c6075554c59ff6ca4bed1c6f82b6786dfef23b0\": rpc error: code = NotFound desc = could not find container \"61af56d59574e2335f02e6aa4c6075554c59ff6ca4bed1c6f82b6786dfef23b0\": container with ID starting with 61af56d59574e2335f02e6aa4c6075554c59ff6ca4bed1c6f82b6786dfef23b0 not found: ID does not exist" Sep 08 04:26:39.045071 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:39.045003 2574 scope.go:117] "RemoveContainer" containerID="8f5126313d4618c4d9615a0c2ec95a43c1c83bf9cf176522a465efd7c925e7c2" Sep 08 04:26:39.045298 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:26:39.045279 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f5126313d4618c4d9615a0c2ec95a43c1c83bf9cf176522a465efd7c925e7c2\": container with ID starting with 8f5126313d4618c4d9615a0c2ec95a43c1c83bf9cf176522a465efd7c925e7c2 not found: ID does not exist" containerID="8f5126313d4618c4d9615a0c2ec95a43c1c83bf9cf176522a465efd7c925e7c2" Sep 08 04:26:39.045342 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:39.045303 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f5126313d4618c4d9615a0c2ec95a43c1c83bf9cf176522a465efd7c925e7c2"} err="failed to get container status \"8f5126313d4618c4d9615a0c2ec95a43c1c83bf9cf176522a465efd7c925e7c2\": rpc error: code = NotFound desc = could not find container \"8f5126313d4618c4d9615a0c2ec95a43c1c83bf9cf176522a465efd7c925e7c2\": container with ID starting with 8f5126313d4618c4d9615a0c2ec95a43c1c83bf9cf176522a465efd7c925e7c2 not found: ID does not exist" Sep 08 04:26:39.050708 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:39.050688 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf"] Sep 08 04:26:39.055593 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:39.055572 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-75264d-openshift-default-56d5f54b78-s7fnf"] Sep 08 04:26:40.567612 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:40.567579 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="078876db-1081-4a8a-bbc4-278589a1f9ad" path="/var/lib/kubelet/pods/078876db-1081-4a8a-bbc4-278589a1f9ad/volumes" Sep 08 04:26:43.884781 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:43.884744 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:26:44.272901 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:44.272869 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:26:44.343013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:44.342976 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:26:44.419295 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:44.419264 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:26:44.472426 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:44.472398 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c2eafd-7t9x2"] Sep 08 04:26:44.477321 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:44.477297 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c2eafd-7t9x2"] Sep 08 04:26:44.495713 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:44.495689 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:26:44.569306 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:44.569221 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc94268c-d21a-49dc-a876-1a33cc55cfff" path="/var/lib/kubelet/pods/dc94268c-d21a-49dc-a876-1a33cc55cfff/volumes" Sep 08 04:26:44.860145 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:44.860054 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:26:50.817271 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:50.817237 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:26:50.917083 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:50.917038 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:26:51.189168 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:26:51.189091 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:27:10.483613 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:10.483572 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:27:38.727457 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:38.727423 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:27:47.482863 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:47.482827 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:27:48.399105 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:48.399076 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:27:48.878978 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:48.878946 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:27:49.196001 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.195929 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk"] Sep 08 04:27:49.196277 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.196265 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc94268c-d21a-49dc-a876-1a33cc55cfff" containerName="revoke-keys" Sep 08 04:27:49.196325 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.196279 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc94268c-d21a-49dc-a876-1a33cc55cfff" containerName="revoke-keys" Sep 08 04:27:49.196325 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.196292 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="078876db-1081-4a8a-bbc4-278589a1f9ad" containerName="istio-proxy" Sep 08 04:27:49.196325 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.196298 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="078876db-1081-4a8a-bbc4-278589a1f9ad" containerName="istio-proxy" Sep 08 04:27:49.196325 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.196310 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="078876db-1081-4a8a-bbc4-278589a1f9ad" containerName="istio-proxy" Sep 08 04:27:49.196325 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.196316 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="078876db-1081-4a8a-bbc4-278589a1f9ad" containerName="istio-proxy" Sep 08 04:27:49.196485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.196373 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="078876db-1081-4a8a-bbc4-278589a1f9ad" containerName="istio-proxy" Sep 08 04:27:49.196485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.196381 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="078876db-1081-4a8a-bbc4-278589a1f9ad" containerName="istio-proxy" Sep 08 04:27:49.196485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.196388 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc94268c-d21a-49dc-a876-1a33cc55cfff" containerName="revoke-keys" Sep 08 04:27:49.199543 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.199521 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.206317 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.206300 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-520856-openshift-default-dockercfg-gmfhz\"" Sep 08 04:27:49.214135 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.214105 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk"] Sep 08 04:27:49.231854 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.231830 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:27:49.303206 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.303170 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-workload-socket\") pod \"e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.303380 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.303209 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istio-envoy\") pod \"e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.303380 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.303296 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-workload-certs\") pod \"e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.303380 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.303350 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istio-data\") pod \"e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.303592 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.303415 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-credential-socket\") pod \"e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.303592 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.303441 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istiod-ca-cert\") pod \"e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.303592 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.303463 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istio-token\") pod \"e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.303592 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.303491 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istio-podinfo\") pod \"e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.303802 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.303601 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrjls\" (UniqueName: \"kubernetes.io/projected/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-kube-api-access-lrjls\") pod \"e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.404640 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.404608 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-workload-certs\") pod \"e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.404773 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.404647 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istio-data\") pod \"e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.404773 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.404674 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-credential-socket\") pod \"e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.404773 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.404700 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istiod-ca-cert\") pod \"e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.404773 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.404731 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istio-token\") pod \"e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.404773 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.404770 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istio-podinfo\") pod \"e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.405019 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.404978 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrjls\" (UniqueName: \"kubernetes.io/projected/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-kube-api-access-lrjls\") pod \"e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.405087 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.405063 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-workload-certs\") pod \"e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.405147 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.405087 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istio-data\") pod \"e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.405147 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.405087 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-workload-socket\") pod \"e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.405147 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.405136 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istio-envoy\") pod \"e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.405321 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.405224 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-credential-socket\") pod \"e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.405389 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.405372 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-workload-socket\") pod \"e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.405595 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.405574 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istiod-ca-cert\") pod \"e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.407405 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.407377 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istio-envoy\") pod \"e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.407496 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.407427 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istio-podinfo\") pod \"e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.413801 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.413779 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrjls\" (UniqueName: \"kubernetes.io/projected/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-kube-api-access-lrjls\") pod \"e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.414121 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.414099 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istio-token\") pod \"e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.509492 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.509470 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:49.539274 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.539250 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:27:49.786913 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.786883 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk"] Sep 08 04:27:49.789014 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:27:49.788988 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dae9cb8_8fe8_482b_a2e5_a9120ea4ef55.slice/crio-d666ab1db82f219998e4ab1741d0f12119caf1ce931c4e30a7e549d0153a4bec WatchSource:0}: Error finding container d666ab1db82f219998e4ab1741d0f12119caf1ce931c4e30a7e549d0153a4bec: Status 404 returned error can't find the container with id d666ab1db82f219998e4ab1741d0f12119caf1ce931c4e30a7e549d0153a4bec Sep 08 04:27:49.791193 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.791163 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:27:49.791253 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.791224 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:27:49.791296 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:49.791254 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:27:50.434446 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:50.434410 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" event={"ID":"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55","Type":"ContainerStarted","Data":"ac4102a274fd5e081b0a4efe72f77af4bc1fe00c19e6dfdd5c934c1147473180"} Sep 08 04:27:50.434822 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:50.434522 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" event={"ID":"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55","Type":"ContainerStarted","Data":"d666ab1db82f219998e4ab1741d0f12119caf1ce931c4e30a7e549d0153a4bec"} Sep 08 04:27:50.457426 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:50.457376 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" podStartSLOduration=1.457362651 podStartE2EDuration="1.457362651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:27:50.455829251 +0000 UTC m=+1644.378290013" watchObservedRunningTime="2026-09-08 04:27:50.457362651 +0000 UTC m=+1644.379823391" Sep 08 04:27:50.510484 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:50.510461 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:50.512182 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:50.512159 2574 patch_prober.go:28] interesting pod/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Sep 08 04:27:50.512282 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:50.512204 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" podUID="8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Sep 08 04:27:51.514497 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:51.514459 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:52.438417 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:52.438386 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:27:52.446050 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:52.446022 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:52.446894 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:52.446876 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:27:52.531179 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:52.531151 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:27:53.697205 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:53.697178 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:27:53.798659 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:53.798625 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:27:53.983389 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:53.983316 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:27:54.238615 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:54.238536 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:27:54.324385 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:54.324351 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6"] Sep 08 04:27:54.326829 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:54.326807 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:27:54.329048 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:54.329032 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6" Sep 08 04:27:54.331793 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:54.331774 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-520856\"" Sep 08 04:27:54.331893 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:54.331791 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-520856-dockercfg-dpzb7\"" Sep 08 04:27:54.341133 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:54.341108 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6"] Sep 08 04:27:54.350394 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:54.350372 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbch7\" (UniqueName: \"kubernetes.io/projected/65066aa3-519f-436f-ae25-5a20d1140232-kube-api-access-lbch7\") pod \"payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6\" (UID: \"65066aa3-519f-436f-ae25-5a20d1140232\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6" Sep 08 04:27:54.350540 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:54.350478 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65066aa3-519f-436f-ae25-5a20d1140232-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6\" (UID: \"65066aa3-519f-436f-ae25-5a20d1140232\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6" Sep 08 04:27:54.451888 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:54.451856 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbch7\" (UniqueName: \"kubernetes.io/projected/65066aa3-519f-436f-ae25-5a20d1140232-kube-api-access-lbch7\") pod \"payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6\" (UID: \"65066aa3-519f-436f-ae25-5a20d1140232\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6" Sep 08 04:27:54.452080 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:54.451953 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65066aa3-519f-436f-ae25-5a20d1140232-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6\" (UID: \"65066aa3-519f-436f-ae25-5a20d1140232\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6" Sep 08 04:27:54.452647 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:54.452626 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65066aa3-519f-436f-ae25-5a20d1140232-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6\" (UID: \"65066aa3-519f-436f-ae25-5a20d1140232\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6" Sep 08 04:27:54.461484 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:54.461452 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbch7\" (UniqueName: \"kubernetes.io/projected/65066aa3-519f-436f-ae25-5a20d1140232-kube-api-access-lbch7\") pod \"payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6\" (UID: \"65066aa3-519f-436f-ae25-5a20d1140232\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6" Sep 08 04:27:54.640902 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:54.640881 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6" Sep 08 04:27:54.916716 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:54.916688 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6"] Sep 08 04:27:54.918166 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:27:54.918141 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65066aa3_519f_436f_ae25_5a20d1140232.slice/crio-f04b8df105a95168f99401daba9627bd0e30abac5bd383f7aac5e1ff55240a5c WatchSource:0}: Error finding container f04b8df105a95168f99401daba9627bd0e30abac5bd383f7aac5e1ff55240a5c: Status 404 returned error can't find the container with id f04b8df105a95168f99401daba9627bd0e30abac5bd383f7aac5e1ff55240a5c Sep 08 04:27:54.966202 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:27:54.966172 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65066aa3_519f_436f_ae25_5a20d1140232.slice/crio-e7c88145e058a5dda7331695c8a0823ad45a0f19df44a017520d02c53e9742e3.scope\": RecentStats: unable to find data in memory cache]" Sep 08 04:27:55.463929 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:55.463897 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6" event={"ID":"65066aa3-519f-436f-ae25-5a20d1140232","Type":"ContainerStarted","Data":"e7c88145e058a5dda7331695c8a0823ad45a0f19df44a017520d02c53e9742e3"} Sep 08 04:27:55.464131 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:55.463945 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6" event={"ID":"65066aa3-519f-436f-ae25-5a20d1140232","Type":"ContainerStarted","Data":"f04b8df105a95168f99401daba9627bd0e30abac5bd383f7aac5e1ff55240a5c"} Sep 08 04:27:55.464131 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:55.464105 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6" Sep 08 04:27:55.482661 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:27:55.482613 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6" podStartSLOduration=1.482600634 podStartE2EDuration="1.482600634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:27:55.481293581 +0000 UTC m=+1649.403754340" watchObservedRunningTime="2026-09-08 04:27:55.482600634 +0000 UTC m=+1649.405061374" Sep 08 04:28:06.473057 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:06.473023 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6" Sep 08 04:28:16.590843 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:16.590804 2574 generic.go:356] "Generic (PLEG): container finished" podID="8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55" containerID="ac4102a274fd5e081b0a4efe72f77af4bc1fe00c19e6dfdd5c934c1147473180" exitCode=0 Sep 08 04:28:16.591277 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:16.590879 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" event={"ID":"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55","Type":"ContainerDied","Data":"ac4102a274fd5e081b0a4efe72f77af4bc1fe00c19e6dfdd5c934c1147473180"} Sep 08 04:28:16.591277 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:16.591267 2574 scope.go:117] "RemoveContainer" containerID="ac4102a274fd5e081b0a4efe72f77af4bc1fe00c19e6dfdd5c934c1147473180" Sep 08 04:28:16.592487 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:16.592452 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:28:16.592588 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:16.592569 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:28:16.592632 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:16.592615 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:28:17.545726 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:17.545685 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:28:17.612080 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:17.612034 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" event={"ID":"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55","Type":"ContainerStarted","Data":"6b6673f7ca737f1f9eb562295445c8b63d2793f8af4804bd0488b16376574f4d"} Sep 08 04:28:18.277115 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.277080 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:28:18.511794 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.511764 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:28:18.512091 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.512068 2574 patch_prober.go:28] interesting pod/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Sep 08 04:28:18.512273 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.512249 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" podUID="8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Sep 08 04:28:18.591552 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.591442 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg"] Sep 08 04:28:18.611947 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.611913 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.618944 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.616659 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg"] Sep 08 04:28:18.618944 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.618125 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-6c84ae-openshift-default-dockercfg-ngwx8\"" Sep 08 04:28:18.623819 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.623795 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:28:18.673008 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.672967 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3427054f-6ed2-482b-817a-8f6859f7c383-istio-token\") pod \"e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.673149 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.673029 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3427054f-6ed2-482b-817a-8f6859f7c383-istiod-ca-cert\") pod \"e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.673149 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.673080 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-workload-certs\") pod \"e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.673290 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.673212 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-credential-socket\") pod \"e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.673354 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.673320 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3427054f-6ed2-482b-817a-8f6859f7c383-istio-podinfo\") pod \"e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.673410 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.673365 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-istio-envoy\") pod \"e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.673410 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.673391 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-workload-socket\") pod \"e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.673553 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.673451 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-istio-data\") pod \"e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.673553 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.673488 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6l4s\" (UniqueName: \"kubernetes.io/projected/3427054f-6ed2-482b-817a-8f6859f7c383-kube-api-access-s6l4s\") pod \"e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.774984 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.774953 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-credential-socket\") pod \"e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.775184 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.775015 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3427054f-6ed2-482b-817a-8f6859f7c383-istio-podinfo\") pod \"e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.775184 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.775047 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-istio-envoy\") pod \"e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.775184 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.775063 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-workload-socket\") pod \"e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.775184 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.775098 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-istio-data\") pod \"e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.775184 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.775130 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6l4s\" (UniqueName: \"kubernetes.io/projected/3427054f-6ed2-482b-817a-8f6859f7c383-kube-api-access-s6l4s\") pod \"e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.775184 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.775171 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3427054f-6ed2-482b-817a-8f6859f7c383-istio-token\") pod \"e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.775567 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.775196 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3427054f-6ed2-482b-817a-8f6859f7c383-istiod-ca-cert\") pod \"e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.775567 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.775225 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-workload-certs\") pod \"e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.775567 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.775424 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-credential-socket\") pod \"e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.775739 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.775562 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-istio-data\") pod \"e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.775739 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.775637 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-workload-certs\") pod \"e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.775739 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.775723 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-workload-socket\") pod \"e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.776138 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.776112 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3427054f-6ed2-482b-817a-8f6859f7c383-istiod-ca-cert\") pod \"e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.777353 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.777332 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-istio-envoy\") pod \"e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.777627 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.777610 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3427054f-6ed2-482b-817a-8f6859f7c383-istio-podinfo\") pod \"e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.783969 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.783950 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6l4s\" (UniqueName: \"kubernetes.io/projected/3427054f-6ed2-482b-817a-8f6859f7c383-kube-api-access-s6l4s\") pod \"e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.784060 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.783984 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3427054f-6ed2-482b-817a-8f6859f7c383-istio-token\") pod \"e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:18.929469 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:18.929402 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:19.255365 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:28:19.255327 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3427054f_6ed2_482b_817a_8f6859f7c383.slice/crio-889766d55d29ce45092b7adcf72eb15de85ac10c9dc2f7b574f586fe8b21b83b WatchSource:0}: Error finding container 889766d55d29ce45092b7adcf72eb15de85ac10c9dc2f7b574f586fe8b21b83b: Status 404 returned error can't find the container with id 889766d55d29ce45092b7adcf72eb15de85ac10c9dc2f7b574f586fe8b21b83b Sep 08 04:28:19.255772 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:19.255736 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg"] Sep 08 04:28:19.257649 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:19.257618 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:28:19.257729 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:19.257679 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:28:19.257729 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:19.257707 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:28:19.287419 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:19.287396 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:28:19.510476 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:19.509946 2574 patch_prober.go:28] interesting pod/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Sep 08 04:28:19.510476 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:19.510020 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" podUID="8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Sep 08 04:28:19.632526 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:19.632470 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" event={"ID":"3427054f-6ed2-482b-817a-8f6859f7c383","Type":"ContainerStarted","Data":"c7b38113168b8384e754f9d8c606b8957e41c3e3ec3b23a922a7ebeac644443f"} Sep 08 04:28:19.632999 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:19.632549 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" event={"ID":"3427054f-6ed2-482b-817a-8f6859f7c383","Type":"ContainerStarted","Data":"889766d55d29ce45092b7adcf72eb15de85ac10c9dc2f7b574f586fe8b21b83b"} Sep 08 04:28:19.656908 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:19.656855 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" podStartSLOduration=1.656838027 podStartE2EDuration="1.656838027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:28:19.653999622 +0000 UTC m=+1673.576460414" watchObservedRunningTime="2026-09-08 04:28:19.656838027 +0000 UTC m=+1673.579298766" Sep 08 04:28:19.930253 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:19.930171 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:20.510552 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:20.510488 2574 patch_prober.go:28] interesting pod/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Sep 08 04:28:20.510765 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:20.510577 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" podUID="8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Sep 08 04:28:20.935228 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:20.935140 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:21.510458 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:21.510408 2574 patch_prober.go:28] interesting pod/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Sep 08 04:28:21.510690 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:21.510480 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" podUID="8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Sep 08 04:28:21.638459 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:21.638421 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:28:21.643161 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:21.643129 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:21.644177 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:21.644160 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:21.734111 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:21.734073 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:28:22.446787 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:22.446754 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:28:22.510065 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:22.510036 2574 patch_prober.go:28] interesting pod/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Sep 08 04:28:22.510235 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:22.510091 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" podUID="8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Sep 08 04:28:22.821094 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:22.821065 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:28:22.916727 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:22.916698 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:28:23.252602 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:23.252572 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:28:23.514346 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:23.514276 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:28:23.515211 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:23.515193 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:28:23.521143 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:23.521122 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:28:23.622728 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:23.622688 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:28:26.905568 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:26.905537 2574 scope.go:117] "RemoveContainer" containerID="627bd03271bdac1a47096fbb3b7c8d0c73d2b7885193331ba5112bd9a6ef6897" Sep 08 04:28:37.684663 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:37.684627 2574 patch_prober.go:28] interesting pod/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 08 04:28:37.685041 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:37.684680 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" podUID="3427054f-6ed2-482b-817a-8f6859f7c383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 08 04:28:38.754890 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:38.754853 2574 generic.go:356] "Generic (PLEG): container finished" podID="3427054f-6ed2-482b-817a-8f6859f7c383" containerID="c7b38113168b8384e754f9d8c606b8957e41c3e3ec3b23a922a7ebeac644443f" exitCode=0 Sep 08 04:28:38.755314 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:38.754926 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" event={"ID":"3427054f-6ed2-482b-817a-8f6859f7c383","Type":"ContainerDied","Data":"c7b38113168b8384e754f9d8c606b8957e41c3e3ec3b23a922a7ebeac644443f"} Sep 08 04:28:38.755416 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:38.755396 2574 scope.go:117] "RemoveContainer" containerID="c7b38113168b8384e754f9d8c606b8957e41c3e3ec3b23a922a7ebeac644443f" Sep 08 04:28:38.756487 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:38.756458 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:28:38.756594 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:38.756579 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:28:38.756636 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:38.756621 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 08 04:28:39.762289 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:39.762253 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" event={"ID":"3427054f-6ed2-482b-817a-8f6859f7c383","Type":"ContainerStarted","Data":"247c845120c619b5bd1e8692569b2aee65cd018f1406a01c6f9137ffaa506b82"} Sep 08 04:28:39.930234 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:39.930197 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:39.930402 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:39.930385 2574 patch_prober.go:28] interesting pod/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Sep 08 04:28:39.930448 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:39.930434 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" podUID="3427054f-6ed2-482b-817a-8f6859f7c383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Sep 08 04:28:40.930715 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:40.930680 2574 patch_prober.go:28] interesting pod/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Sep 08 04:28:40.931078 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:40.930744 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" podUID="3427054f-6ed2-482b-817a-8f6859f7c383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Sep 08 04:28:41.930444 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:41.930414 2574 patch_prober.go:28] interesting pod/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Sep 08 04:28:41.930637 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:41.930485 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" podUID="3427054f-6ed2-482b-817a-8f6859f7c383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Sep 08 04:28:42.145387 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:42.145342 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:28:42.930742 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:42.930704 2574 patch_prober.go:28] interesting pod/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Sep 08 04:28:42.930914 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:42.930768 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" podUID="3427054f-6ed2-482b-817a-8f6859f7c383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Sep 08 04:28:43.930493 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:43.930461 2574 patch_prober.go:28] interesting pod/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Sep 08 04:28:43.930991 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:43.930543 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" podUID="3427054f-6ed2-482b-817a-8f6859f7c383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Sep 08 04:28:44.933009 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:44.932980 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:44.933495 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:44.933404 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:44.934016 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:44.933995 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:28:46.690126 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:28:46.690089 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:29:25.216990 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:29:25.216955 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:29:25.283122 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:29:25.283091 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:29:27.233948 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:29:27.233892 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:29:27.336269 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:29:27.336241 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:29:33.053376 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:29:33.053334 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:29:55.679522 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:29:55.679472 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:30:00.144880 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:00.144842 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-720789-29814030-k4p6r"] Sep 08 04:30:00.148648 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:00.148626 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814030-kxj7d"] Sep 08 04:30:00.148778 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:00.148745 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-720789-29814030-k4p6r" Sep 08 04:30:00.151196 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:00.151177 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7r29w\"" Sep 08 04:30:00.151766 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:00.151751 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2a8bb-29814030-f8gcd"] Sep 08 04:30:00.151877 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:00.151856 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814030-kxj7d" Sep 08 04:30:00.155095 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:00.155071 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6c84ae-29814030-m8vnz"] Sep 08 04:30:00.155209 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:00.155181 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2a8bb-29814030-f8gcd" Sep 08 04:30:00.162403 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:00.161670 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6c84ae-29814030-m8vnz" Sep 08 04:30:00.162403 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:00.162252 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-720789-29814030-k4p6r" Sep 08 04:30:00.170673 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:00.170645 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814030-kxj7d"] Sep 08 04:30:00.178452 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:00.178424 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814030-kxj7d" Sep 08 04:30:00.183817 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:00.183786 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-720789-29814030-k4p6r"] Sep 08 04:30:00.188839 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:00.188817 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2a8bb-29814030-f8gcd"] Sep 08 04:30:00.192135 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:00.192066 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6c84ae-29814030-m8vnz"] Sep 08 04:30:00.226062 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:00.225906 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2a8bb-29814030-f8gcd" Sep 08 04:30:00.232671 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:00.232476 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6c84ae-29814030-m8vnz" Sep 08 04:30:00.460362 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:00.460332 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-720789-29814030-k4p6r"] Sep 08 04:30:00.461258 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:30:00.461214 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeca0803_5381_411b_bda9_ede149a440e7.slice/crio-8952770dd150d346dc38a0026b9b35c6e54d151db69e1b432805507f92801099 WatchSource:0}: Error finding container 8952770dd150d346dc38a0026b9b35c6e54d151db69e1b432805507f92801099: Status 404 returned error can't find the container with id 8952770dd150d346dc38a0026b9b35c6e54d151db69e1b432805507f92801099 Sep 08 04:30:00.478740 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:00.478716 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814030-kxj7d"] Sep 08 04:30:00.480728 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:30:00.480703 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2eb5f92_6531_4ef5_9d47_4b50d0105454.slice/crio-4aea670384d044042a01f72e0eb8aeb27b79af87b8e44a5b5302ab40073febad WatchSource:0}: Error finding container 4aea670384d044042a01f72e0eb8aeb27b79af87b8e44a5b5302ab40073febad: Status 404 returned error can't find the container with id 4aea670384d044042a01f72e0eb8aeb27b79af87b8e44a5b5302ab40073febad Sep 08 04:30:00.505892 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:00.505857 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6_65066aa3-519f-436f-ae25-5a20d1140232/payload-processing/0.log" Sep 08 04:30:00.524187 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:00.524158 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2a8bb-29814030-f8gcd"] Sep 08 04:30:00.525240 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:30:00.525203 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6207719_c7aa_45a8_a2cb_8eb7ed081c76.slice/crio-da34955646d5258058359f09074ba78130d9df6209f4933a41e9af02b19d4157 WatchSource:0}: Error finding container da34955646d5258058359f09074ba78130d9df6209f4933a41e9af02b19d4157: Status 404 returned error can't find the container with id da34955646d5258058359f09074ba78130d9df6209f4933a41e9af02b19d4157 Sep 08 04:30:00.543817 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:00.543793 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6c84ae-29814030-m8vnz"] Sep 08 04:30:00.545671 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:30:00.545641 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e4ad2f_2781_469f_ab11_211451e914b5.slice/crio-d2dba25c7f357a8cf53031e0e21ce9fcb7ea94e1a5744c8a5329bd3b8c496973 WatchSource:0}: Error finding container d2dba25c7f357a8cf53031e0e21ce9fcb7ea94e1a5744c8a5329bd3b8c496973: Status 404 returned error can't find the container with id d2dba25c7f357a8cf53031e0e21ce9fcb7ea94e1a5744c8a5329bd3b8c496973 Sep 08 04:30:01.031621 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:01.031592 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:30:01.243321 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:01.243286 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2a8bb-29814030-f8gcd" event={"ID":"d6207719-c7aa-45a8-a2cb-8eb7ed081c76","Type":"ContainerStarted","Data":"411fb75597dfac1b3dd6ac86a7d3c91e4f182df074b48c94069e134ba60c9113"} Sep 08 04:30:01.243806 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:01.243349 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2a8bb-29814030-f8gcd" event={"ID":"d6207719-c7aa-45a8-a2cb-8eb7ed081c76","Type":"ContainerStarted","Data":"da34955646d5258058359f09074ba78130d9df6209f4933a41e9af02b19d4157"} Sep 08 04:30:01.245027 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:01.244993 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814030-kxj7d" event={"ID":"d2eb5f92-6531-4ef5-9d47-4b50d0105454","Type":"ContainerStarted","Data":"485fe46715a317e5a5f956bb1ed83411e23ec547225df53e87981f1e45050a4e"} Sep 08 04:30:01.245150 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:01.245040 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814030-kxj7d" event={"ID":"d2eb5f92-6531-4ef5-9d47-4b50d0105454","Type":"ContainerStarted","Data":"4aea670384d044042a01f72e0eb8aeb27b79af87b8e44a5b5302ab40073febad"} Sep 08 04:30:01.246462 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:01.246430 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-720789-29814030-k4p6r" event={"ID":"aeca0803-5381-411b-bda9-ede149a440e7","Type":"ContainerStarted","Data":"f09a5564902036f139addb456ba8dae8a30d9d298fb5428a18f60b2a557f5574"} Sep 08 04:30:01.246585 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:01.246496 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-720789-29814030-k4p6r" event={"ID":"aeca0803-5381-411b-bda9-ede149a440e7","Type":"ContainerStarted","Data":"8952770dd150d346dc38a0026b9b35c6e54d151db69e1b432805507f92801099"} Sep 08 04:30:01.247954 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:01.247933 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6c84ae-29814030-m8vnz" event={"ID":"44e4ad2f-2781-469f-ab11-211451e914b5","Type":"ContainerStarted","Data":"d02d42dd38d2ccaa8b44b6f07ab7f6631842ca65bf27a8ce20529585bd37d21c"} Sep 08 04:30:01.248042 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:01.247971 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6c84ae-29814030-m8vnz" event={"ID":"44e4ad2f-2781-469f-ab11-211451e914b5","Type":"ContainerStarted","Data":"d2dba25c7f357a8cf53031e0e21ce9fcb7ea94e1a5744c8a5329bd3b8c496973"} Sep 08 04:30:01.261922 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:01.261847 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2a8bb-29814030-f8gcd" podStartSLOduration=1.261831485 podStartE2EDuration="1.261831485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:30:01.259769227 +0000 UTC m=+1775.182229991" watchObservedRunningTime="2026-09-08 04:30:01.261831485 +0000 UTC m=+1775.184292224" Sep 08 04:30:01.279865 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:01.279817 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814030-kxj7d" podStartSLOduration=1.27980075 podStartE2EDuration="1.27980075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:30:01.278345242 +0000 UTC m=+1775.200806008" watchObservedRunningTime="2026-09-08 04:30:01.27980075 +0000 UTC m=+1775.202261489" Sep 08 04:30:01.297552 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:01.297426 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-720789-29814030-k4p6r" podStartSLOduration=1.297410015 podStartE2EDuration="1.297410015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:30:01.294563632 +0000 UTC m=+1775.217024393" watchObservedRunningTime="2026-09-08 04:30:01.297410015 +0000 UTC m=+1775.219870806" Sep 08 04:30:01.315193 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:01.315134 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6c84ae-29814030-m8vnz" podStartSLOduration=1.315109283 podStartE2EDuration="1.315109283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:30:01.313898185 +0000 UTC m=+1775.236358937" watchObservedRunningTime="2026-09-08 04:30:01.315109283 +0000 UTC m=+1775.237570023" Sep 08 04:30:01.366342 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:01.366311 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:30:01.681384 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:30:01.681345 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6207719_c7aa_45a8_a2cb_8eb7ed081c76.slice/crio-411fb75597dfac1b3dd6ac86a7d3c91e4f182df074b48c94069e134ba60c9113.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6207719_c7aa_45a8_a2cb_8eb7ed081c76.slice/crio-conmon-411fb75597dfac1b3dd6ac86a7d3c91e4f182df074b48c94069e134ba60c9113.scope\": RecentStats: unable to find data in memory cache]" Sep 08 04:30:01.808761 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:01.808730 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:30:02.261028 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:02.260993 2574 generic.go:356] "Generic (PLEG): container finished" podID="aeca0803-5381-411b-bda9-ede149a440e7" containerID="f09a5564902036f139addb456ba8dae8a30d9d298fb5428a18f60b2a557f5574" exitCode=0 Sep 08 04:30:02.261462 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:02.261081 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-720789-29814030-k4p6r" event={"ID":"aeca0803-5381-411b-bda9-ede149a440e7","Type":"ContainerDied","Data":"f09a5564902036f139addb456ba8dae8a30d9d298fb5428a18f60b2a557f5574"} Sep 08 04:30:02.262945 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:02.262917 2574 generic.go:356] "Generic (PLEG): container finished" podID="44e4ad2f-2781-469f-ab11-211451e914b5" containerID="d02d42dd38d2ccaa8b44b6f07ab7f6631842ca65bf27a8ce20529585bd37d21c" exitCode=0 Sep 08 04:30:02.263051 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:02.262999 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6c84ae-29814030-m8vnz" event={"ID":"44e4ad2f-2781-469f-ab11-211451e914b5","Type":"ContainerDied","Data":"d02d42dd38d2ccaa8b44b6f07ab7f6631842ca65bf27a8ce20529585bd37d21c"} Sep 08 04:30:02.264634 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:02.264612 2574 generic.go:356] "Generic (PLEG): container finished" podID="d6207719-c7aa-45a8-a2cb-8eb7ed081c76" containerID="411fb75597dfac1b3dd6ac86a7d3c91e4f182df074b48c94069e134ba60c9113" exitCode=0 Sep 08 04:30:02.264734 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:02.264691 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2a8bb-29814030-f8gcd" event={"ID":"d6207719-c7aa-45a8-a2cb-8eb7ed081c76","Type":"ContainerDied","Data":"411fb75597dfac1b3dd6ac86a7d3c91e4f182df074b48c94069e134ba60c9113"} Sep 08 04:30:02.266170 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:02.266147 2574 generic.go:356] "Generic (PLEG): container finished" podID="d2eb5f92-6531-4ef5-9d47-4b50d0105454" containerID="485fe46715a317e5a5f956bb1ed83411e23ec547225df53e87981f1e45050a4e" exitCode=0 Sep 08 04:30:02.266274 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:02.266170 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814030-kxj7d" event={"ID":"d2eb5f92-6531-4ef5-9d47-4b50d0105454","Type":"ContainerDied","Data":"485fe46715a317e5a5f956bb1ed83411e23ec547225df53e87981f1e45050a4e"} Sep 08 04:30:03.434625 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:03.433764 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-720789-29814030-k4p6r" Sep 08 04:30:03.502102 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:03.502070 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2a8bb-29814030-f8gcd" Sep 08 04:30:03.505274 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:03.505243 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814030-kxj7d" Sep 08 04:30:03.512277 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:03.512259 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6c84ae-29814030-m8vnz" Sep 08 04:30:04.279583 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:04.279553 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2a8bb-29814030-f8gcd" Sep 08 04:30:04.279583 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:04.279575 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2a8bb-29814030-f8gcd" event={"ID":"d6207719-c7aa-45a8-a2cb-8eb7ed081c76","Type":"ContainerDied","Data":"da34955646d5258058359f09074ba78130d9df6209f4933a41e9af02b19d4157"} Sep 08 04:30:04.279838 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:04.279630 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da34955646d5258058359f09074ba78130d9df6209f4933a41e9af02b19d4157" Sep 08 04:30:04.280802 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:04.280781 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814030-kxj7d" event={"ID":"d2eb5f92-6531-4ef5-9d47-4b50d0105454","Type":"ContainerDied","Data":"4aea670384d044042a01f72e0eb8aeb27b79af87b8e44a5b5302ab40073febad"} Sep 08 04:30:04.280910 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:04.280821 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4aea670384d044042a01f72e0eb8aeb27b79af87b8e44a5b5302ab40073febad" Sep 08 04:30:04.280910 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:04.280785 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814030-kxj7d" Sep 08 04:30:04.282118 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:04.282092 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-720789-29814030-k4p6r" Sep 08 04:30:04.282205 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:04.282140 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-720789-29814030-k4p6r" event={"ID":"aeca0803-5381-411b-bda9-ede149a440e7","Type":"ContainerDied","Data":"8952770dd150d346dc38a0026b9b35c6e54d151db69e1b432805507f92801099"} Sep 08 04:30:04.282205 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:04.282181 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8952770dd150d346dc38a0026b9b35c6e54d151db69e1b432805507f92801099" Sep 08 04:30:04.283485 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:04.283449 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6c84ae-29814030-m8vnz" event={"ID":"44e4ad2f-2781-469f-ab11-211451e914b5","Type":"ContainerDied","Data":"d2dba25c7f357a8cf53031e0e21ce9fcb7ea94e1a5744c8a5329bd3b8c496973"} Sep 08 04:30:04.283591 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:04.283494 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2dba25c7f357a8cf53031e0e21ce9fcb7ea94e1a5744c8a5329bd3b8c496973" Sep 08 04:30:04.283591 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:04.283542 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6c84ae-29814030-m8vnz" Sep 08 04:30:04.825565 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:04.825533 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:30:04.919946 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:04.919914 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:30:06.036381 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:06.036349 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:30:06.136009 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:06.135970 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:30:06.386266 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:06.386185 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:30:06.851154 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:06.851121 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:30:06.868232 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:06.868200 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8"] Sep 08 04:30:06.868588 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:06.868574 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2eb5f92-6531-4ef5-9d47-4b50d0105454" containerName="cleanup" Sep 08 04:30:06.868643 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:06.868590 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2eb5f92-6531-4ef5-9d47-4b50d0105454" containerName="cleanup" Sep 08 04:30:06.868643 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:06.868610 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44e4ad2f-2781-469f-ab11-211451e914b5" containerName="cleanup" Sep 08 04:30:06.868643 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:06.868617 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="44e4ad2f-2781-469f-ab11-211451e914b5" containerName="cleanup" Sep 08 04:30:06.868643 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:06.868628 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeca0803-5381-411b-bda9-ede149a440e7" containerName="cleanup" Sep 08 04:30:06.868643 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:06.868633 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeca0803-5381-411b-bda9-ede149a440e7" containerName="cleanup" Sep 08 04:30:06.868643 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:06.868640 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6207719-c7aa-45a8-a2cb-8eb7ed081c76" containerName="cleanup" Sep 08 04:30:06.868643 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:06.868645 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6207719-c7aa-45a8-a2cb-8eb7ed081c76" containerName="cleanup" Sep 08 04:30:06.868853 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:06.868697 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="44e4ad2f-2781-469f-ab11-211451e914b5" containerName="cleanup" Sep 08 04:30:06.868853 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:06.868704 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6207719-c7aa-45a8-a2cb-8eb7ed081c76" containerName="cleanup" Sep 08 04:30:06.868853 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:06.868713 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2eb5f92-6531-4ef5-9d47-4b50d0105454" containerName="cleanup" Sep 08 04:30:06.868853 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:06.868719 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeca0803-5381-411b-bda9-ede149a440e7" containerName="cleanup" Sep 08 04:30:06.871971 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:06.871955 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8" Sep 08 04:30:06.874563 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:06.874539 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-431421\"" Sep 08 04:30:06.874689 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:06.874568 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-431421-dockercfg-pmp9p\"" Sep 08 04:30:06.881660 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:06.881569 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8"] Sep 08 04:30:06.946319 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:06.946269 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:30:06.950149 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:06.950124 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98"] Sep 08 04:30:06.953744 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:06.953718 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98" Sep 08 04:30:06.962985 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:06.962955 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn8fp\" (UniqueName: \"kubernetes.io/projected/d5755222-f4ee-4a59-a270-3f49144fbe1c-kube-api-access-nn8fp\") pod \"payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8\" (UID: \"d5755222-f4ee-4a59-a270-3f49144fbe1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8" Sep 08 04:30:06.963128 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:06.962997 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5755222-f4ee-4a59-a270-3f49144fbe1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8\" (UID: \"d5755222-f4ee-4a59-a270-3f49144fbe1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8" Sep 08 04:30:06.964443 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:06.964420 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98"] Sep 08 04:30:07.064418 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:07.064378 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn8fp\" (UniqueName: \"kubernetes.io/projected/d5755222-f4ee-4a59-a270-3f49144fbe1c-kube-api-access-nn8fp\") pod \"payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8\" (UID: \"d5755222-f4ee-4a59-a270-3f49144fbe1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8" Sep 08 04:30:07.065037 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:07.065006 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5755222-f4ee-4a59-a270-3f49144fbe1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8\" (UID: \"d5755222-f4ee-4a59-a270-3f49144fbe1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8" Sep 08 04:30:07.065170 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:07.065073 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf4d06a-d520-498e-a748-06daf5d53fdb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98\" (UID: \"abf4d06a-d520-498e-a748-06daf5d53fdb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98" Sep 08 04:30:07.065170 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:07.065101 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5nbc\" (UniqueName: \"kubernetes.io/projected/abf4d06a-d520-498e-a748-06daf5d53fdb-kube-api-access-z5nbc\") pod \"payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98\" (UID: \"abf4d06a-d520-498e-a748-06daf5d53fdb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98" Sep 08 04:30:07.065992 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:07.065968 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5755222-f4ee-4a59-a270-3f49144fbe1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8\" (UID: \"d5755222-f4ee-4a59-a270-3f49144fbe1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8" Sep 08 04:30:07.072438 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:07.072408 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn8fp\" (UniqueName: \"kubernetes.io/projected/d5755222-f4ee-4a59-a270-3f49144fbe1c-kube-api-access-nn8fp\") pod \"payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8\" (UID: \"d5755222-f4ee-4a59-a270-3f49144fbe1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8" Sep 08 04:30:07.166779 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:07.166693 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf4d06a-d520-498e-a748-06daf5d53fdb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98\" (UID: \"abf4d06a-d520-498e-a748-06daf5d53fdb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98" Sep 08 04:30:07.166779 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:07.166735 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5nbc\" (UniqueName: \"kubernetes.io/projected/abf4d06a-d520-498e-a748-06daf5d53fdb-kube-api-access-z5nbc\") pod \"payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98\" (UID: \"abf4d06a-d520-498e-a748-06daf5d53fdb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98" Sep 08 04:30:07.167420 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:07.167401 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf4d06a-d520-498e-a748-06daf5d53fdb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98\" (UID: \"abf4d06a-d520-498e-a748-06daf5d53fdb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98" Sep 08 04:30:07.178035 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:07.178004 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5nbc\" (UniqueName: \"kubernetes.io/projected/abf4d06a-d520-498e-a748-06daf5d53fdb-kube-api-access-z5nbc\") pod \"payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98\" (UID: \"abf4d06a-d520-498e-a748-06daf5d53fdb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98" Sep 08 04:30:07.182857 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:07.182835 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8" Sep 08 04:30:07.265614 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:07.265571 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98" Sep 08 04:30:07.465268 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:07.465242 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8"] Sep 08 04:30:07.466562 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:30:07.466536 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5755222_f4ee_4a59_a270_3f49144fbe1c.slice/crio-187c67bf3d7339b2eb5cffc56e287bf7da65f3b130e76ab873d07174b046e1a6 WatchSource:0}: Error finding container 187c67bf3d7339b2eb5cffc56e287bf7da65f3b130e76ab873d07174b046e1a6: Status 404 returned error can't find the container with id 187c67bf3d7339b2eb5cffc56e287bf7da65f3b130e76ab873d07174b046e1a6 Sep 08 04:30:07.549294 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:07.549264 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98"] Sep 08 04:30:07.550165 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:30:07.550132 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf4d06a_d520_498e_a748_06daf5d53fdb.slice/crio-1cec5450f2ad00dba22ac563566b5995baf998bc727d2fc57d24bb15e9631b61 WatchSource:0}: Error finding container 1cec5450f2ad00dba22ac563566b5995baf998bc727d2fc57d24bb15e9631b61: Status 404 returned error can't find the container with id 1cec5450f2ad00dba22ac563566b5995baf998bc727d2fc57d24bb15e9631b61 Sep 08 04:30:08.310367 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:08.310333 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8" event={"ID":"d5755222-f4ee-4a59-a270-3f49144fbe1c","Type":"ContainerStarted","Data":"208c24f45d25e8496ddbf4ce1e820b80deffb9e38521805c3aaf69a80780dcc9"} Sep 08 04:30:08.310844 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:08.310386 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8" event={"ID":"d5755222-f4ee-4a59-a270-3f49144fbe1c","Type":"ContainerStarted","Data":"187c67bf3d7339b2eb5cffc56e287bf7da65f3b130e76ab873d07174b046e1a6"} Sep 08 04:30:08.310844 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:08.310446 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8" Sep 08 04:30:08.312128 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:08.312098 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98" event={"ID":"abf4d06a-d520-498e-a748-06daf5d53fdb","Type":"ContainerStarted","Data":"78879d92b6e400f09e0846fc52ef4cd604062eb673cc0bc7664cd6a0fb44f33d"} Sep 08 04:30:08.312250 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:08.312153 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98" Sep 08 04:30:08.312250 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:08.312177 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98" event={"ID":"abf4d06a-d520-498e-a748-06daf5d53fdb","Type":"ContainerStarted","Data":"1cec5450f2ad00dba22ac563566b5995baf998bc727d2fc57d24bb15e9631b61"} Sep 08 04:30:08.330183 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:08.330142 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8" podStartSLOduration=2.330128445 podStartE2EDuration="2.330128445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:30:08.326609943 +0000 UTC m=+1782.249070703" watchObservedRunningTime="2026-09-08 04:30:08.330128445 +0000 UTC m=+1782.252589185" Sep 08 04:30:08.346419 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:08.346362 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98" podStartSLOduration=2.34634708 podStartE2EDuration="2.34634708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:30:08.343195025 +0000 UTC m=+1782.265655785" watchObservedRunningTime="2026-09-08 04:30:08.34634708 +0000 UTC m=+1782.268807819" Sep 08 04:30:15.684672 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:15.684640 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:30:19.320160 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:19.320077 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8" Sep 08 04:30:19.320527 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:19.320311 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98" Sep 08 04:30:24.452764 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:24.452727 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:30:24.568539 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:24.567301 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:30:26.428035 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:26.428000 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:30:27.491946 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:27.491915 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:30:29.755888 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:29.755847 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:30:30.451309 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:30.451274 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-431421-vf9v2"] Sep 08 04:30:30.454564 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:30.454547 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-431421-vf9v2" Sep 08 04:30:30.457190 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:30.457166 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 04:30:30.457334 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:30.457193 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7r29w\"" Sep 08 04:30:30.461798 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:30.461775 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-431421-vf9v2"] Sep 08 04:30:30.584057 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:30.584020 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa2217f9-c043-44f9-bf4e-3130deae06ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-431421-vf9v2\" (UID: \"fa2217f9-c043-44f9-bf4e-3130deae06ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-431421-vf9v2" Sep 08 04:30:30.684750 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:30.684715 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa2217f9-c043-44f9-bf4e-3130deae06ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-431421-vf9v2\" (UID: \"fa2217f9-c043-44f9-bf4e-3130deae06ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-431421-vf9v2" Sep 08 04:30:30.685398 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:30.685378 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa2217f9-c043-44f9-bf4e-3130deae06ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-431421-vf9v2\" (UID: \"fa2217f9-c043-44f9-bf4e-3130deae06ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-431421-vf9v2" Sep 08 04:30:30.766266 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:30.766245 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-431421-vf9v2" Sep 08 04:30:31.045861 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:31.045838 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-431421-vf9v2"] Sep 08 04:30:31.047357 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:30:31.047328 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa2217f9_c043_44f9_bf4e_3130deae06ae.slice/crio-fc758dc97090951ecdff10c28785d826c3e346594b7b4f76e2e395a073bd176a WatchSource:0}: Error finding container fc758dc97090951ecdff10c28785d826c3e346594b7b4f76e2e395a073bd176a: Status 404 returned error can't find the container with id fc758dc97090951ecdff10c28785d826c3e346594b7b4f76e2e395a073bd176a Sep 08 04:30:31.448398 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:31.448358 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-431421-vf9v2" event={"ID":"fa2217f9-c043-44f9-bf4e-3130deae06ae","Type":"ContainerStarted","Data":"28ab9c58386fa66c10837e8f452e10f97c4f64bb3fae2377b737953d33d05d5d"} Sep 08 04:30:31.448605 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:31.448417 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-431421-vf9v2" event={"ID":"fa2217f9-c043-44f9-bf4e-3130deae06ae","Type":"ContainerStarted","Data":"fc758dc97090951ecdff10c28785d826c3e346594b7b4f76e2e395a073bd176a"} Sep 08 04:30:31.465910 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:31.465858 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-431421-vf9v2" podStartSLOduration=1.465842199 podStartE2EDuration="1.465842199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:30:31.463610616 +0000 UTC m=+1805.386071373" watchObservedRunningTime="2026-09-08 04:30:31.465842199 +0000 UTC m=+1805.388302939" Sep 08 04:30:32.464523 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:32.464472 2574 generic.go:356] "Generic (PLEG): container finished" podID="fa2217f9-c043-44f9-bf4e-3130deae06ae" containerID="28ab9c58386fa66c10837e8f452e10f97c4f64bb3fae2377b737953d33d05d5d" exitCode=0 Sep 08 04:30:32.464912 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:32.464551 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-431421-vf9v2" event={"ID":"fa2217f9-c043-44f9-bf4e-3130deae06ae","Type":"ContainerDied","Data":"28ab9c58386fa66c10837e8f452e10f97c4f64bb3fae2377b737953d33d05d5d"} Sep 08 04:30:33.599695 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:33.599671 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-431421-vf9v2" Sep 08 04:30:33.713465 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:33.713432 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa2217f9-c043-44f9-bf4e-3130deae06ae-maas-api-service-ca\") pod \"fa2217f9-c043-44f9-bf4e-3130deae06ae\" (UID: \"fa2217f9-c043-44f9-bf4e-3130deae06ae\") " Sep 08 04:30:33.713795 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:33.713772 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2217f9-c043-44f9-bf4e-3130deae06ae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fa2217f9-c043-44f9-bf4e-3130deae06ae" (UID: "fa2217f9-c043-44f9-bf4e-3130deae06ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:30:33.713879 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:33.713857 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa2217f9-c043-44f9-bf4e-3130deae06ae-maas-api-service-ca\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:30:34.478900 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:34.478870 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-431421-vf9v2" Sep 08 04:30:34.478900 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:34.478897 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-431421-vf9v2" event={"ID":"fa2217f9-c043-44f9-bf4e-3130deae06ae","Type":"ContainerDied","Data":"fc758dc97090951ecdff10c28785d826c3e346594b7b4f76e2e395a073bd176a"} Sep 08 04:30:34.479115 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:34.478942 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc758dc97090951ecdff10c28785d826c3e346594b7b4f76e2e395a073bd176a" Sep 08 04:30:37.062419 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.062379 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:30:37.097780 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.097745 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98"] Sep 08 04:30:37.098020 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.097992 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98" podUID="abf4d06a-d520-498e-a748-06daf5d53fdb" containerName="payload-processing" containerID="cri-o://78879d92b6e400f09e0846fc52ef4cd604062eb673cc0bc7664cd6a0fb44f33d" gracePeriod=30 Sep 08 04:30:37.107748 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.107724 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8"] Sep 08 04:30:37.108121 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.108092 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8" podUID="d5755222-f4ee-4a59-a270-3f49144fbe1c" containerName="payload-pre-processing" containerID="cri-o://208c24f45d25e8496ddbf4ce1e820b80deffb9e38521805c3aaf69a80780dcc9" gracePeriod=30 Sep 08 04:30:37.152409 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.152359 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:30:37.230869 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.230826 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:30:37.297135 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.297099 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-431421-vf9v2"] Sep 08 04:30:37.299480 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.299456 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:30:37.301887 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.301864 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-431421-vf9v2"] Sep 08 04:30:37.351589 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.351561 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98" Sep 08 04:30:37.382180 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.382147 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:30:37.385830 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.385812 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8" Sep 08 04:30:37.454778 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.454744 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf4d06a-d520-498e-a748-06daf5d53fdb-plugins-config-volume\") pod \"abf4d06a-d520-498e-a748-06daf5d53fdb\" (UID: \"abf4d06a-d520-498e-a748-06daf5d53fdb\") " Sep 08 04:30:37.454966 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.454903 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5nbc\" (UniqueName: \"kubernetes.io/projected/abf4d06a-d520-498e-a748-06daf5d53fdb-kube-api-access-z5nbc\") pod \"abf4d06a-d520-498e-a748-06daf5d53fdb\" (UID: \"abf4d06a-d520-498e-a748-06daf5d53fdb\") " Sep 08 04:30:37.455244 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.455203 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf4d06a-d520-498e-a748-06daf5d53fdb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "abf4d06a-d520-498e-a748-06daf5d53fdb" (UID: "abf4d06a-d520-498e-a748-06daf5d53fdb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:30:37.457013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.456983 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf4d06a-d520-498e-a748-06daf5d53fdb-kube-api-access-z5nbc" (OuterVolumeSpecName: "kube-api-access-z5nbc") pod "abf4d06a-d520-498e-a748-06daf5d53fdb" (UID: "abf4d06a-d520-498e-a748-06daf5d53fdb"). InnerVolumeSpecName "kube-api-access-z5nbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:30:37.499449 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.499417 2574 generic.go:356] "Generic (PLEG): container finished" podID="d5755222-f4ee-4a59-a270-3f49144fbe1c" containerID="208c24f45d25e8496ddbf4ce1e820b80deffb9e38521805c3aaf69a80780dcc9" exitCode=0 Sep 08 04:30:37.499646 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.499464 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8" Sep 08 04:30:37.499646 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.499498 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8" event={"ID":"d5755222-f4ee-4a59-a270-3f49144fbe1c","Type":"ContainerDied","Data":"208c24f45d25e8496ddbf4ce1e820b80deffb9e38521805c3aaf69a80780dcc9"} Sep 08 04:30:37.499646 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.499610 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8" event={"ID":"d5755222-f4ee-4a59-a270-3f49144fbe1c","Type":"ContainerDied","Data":"187c67bf3d7339b2eb5cffc56e287bf7da65f3b130e76ab873d07174b046e1a6"} Sep 08 04:30:37.499646 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.499639 2574 scope.go:117] "RemoveContainer" containerID="208c24f45d25e8496ddbf4ce1e820b80deffb9e38521805c3aaf69a80780dcc9" Sep 08 04:30:37.500955 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.500937 2574 generic.go:356] "Generic (PLEG): container finished" podID="abf4d06a-d520-498e-a748-06daf5d53fdb" containerID="78879d92b6e400f09e0846fc52ef4cd604062eb673cc0bc7664cd6a0fb44f33d" exitCode=0 Sep 08 04:30:37.501050 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.500968 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98" event={"ID":"abf4d06a-d520-498e-a748-06daf5d53fdb","Type":"ContainerDied","Data":"78879d92b6e400f09e0846fc52ef4cd604062eb673cc0bc7664cd6a0fb44f33d"} Sep 08 04:30:37.501050 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.500988 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98" Sep 08 04:30:37.501050 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.501017 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98" event={"ID":"abf4d06a-d520-498e-a748-06daf5d53fdb","Type":"ContainerDied","Data":"1cec5450f2ad00dba22ac563566b5995baf998bc727d2fc57d24bb15e9631b61"} Sep 08 04:30:37.509440 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.509423 2574 scope.go:117] "RemoveContainer" containerID="208c24f45d25e8496ddbf4ce1e820b80deffb9e38521805c3aaf69a80780dcc9" Sep 08 04:30:37.509727 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:30:37.509708 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"208c24f45d25e8496ddbf4ce1e820b80deffb9e38521805c3aaf69a80780dcc9\": container with ID starting with 208c24f45d25e8496ddbf4ce1e820b80deffb9e38521805c3aaf69a80780dcc9 not found: ID does not exist" containerID="208c24f45d25e8496ddbf4ce1e820b80deffb9e38521805c3aaf69a80780dcc9" Sep 08 04:30:37.509816 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.509738 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"208c24f45d25e8496ddbf4ce1e820b80deffb9e38521805c3aaf69a80780dcc9"} err="failed to get container status \"208c24f45d25e8496ddbf4ce1e820b80deffb9e38521805c3aaf69a80780dcc9\": rpc error: code = NotFound desc = could not find container \"208c24f45d25e8496ddbf4ce1e820b80deffb9e38521805c3aaf69a80780dcc9\": container with ID starting with 208c24f45d25e8496ddbf4ce1e820b80deffb9e38521805c3aaf69a80780dcc9 not found: ID does not exist" Sep 08 04:30:37.509816 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.509759 2574 scope.go:117] "RemoveContainer" containerID="78879d92b6e400f09e0846fc52ef4cd604062eb673cc0bc7664cd6a0fb44f33d" Sep 08 04:30:37.517865 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.517844 2574 scope.go:117] "RemoveContainer" containerID="78879d92b6e400f09e0846fc52ef4cd604062eb673cc0bc7664cd6a0fb44f33d" Sep 08 04:30:37.518119 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:30:37.518098 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78879d92b6e400f09e0846fc52ef4cd604062eb673cc0bc7664cd6a0fb44f33d\": container with ID starting with 78879d92b6e400f09e0846fc52ef4cd604062eb673cc0bc7664cd6a0fb44f33d not found: ID does not exist" containerID="78879d92b6e400f09e0846fc52ef4cd604062eb673cc0bc7664cd6a0fb44f33d" Sep 08 04:30:37.518181 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.518125 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78879d92b6e400f09e0846fc52ef4cd604062eb673cc0bc7664cd6a0fb44f33d"} err="failed to get container status \"78879d92b6e400f09e0846fc52ef4cd604062eb673cc0bc7664cd6a0fb44f33d\": rpc error: code = NotFound desc = could not find container \"78879d92b6e400f09e0846fc52ef4cd604062eb673cc0bc7664cd6a0fb44f33d\": container with ID starting with 78879d92b6e400f09e0846fc52ef4cd604062eb673cc0bc7664cd6a0fb44f33d not found: ID does not exist" Sep 08 04:30:37.525140 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.525120 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98"] Sep 08 04:30:37.529207 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.529188 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-431421-68c5b44989-dfq98"] Sep 08 04:30:37.555740 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.555719 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nn8fp\" (UniqueName: \"kubernetes.io/projected/d5755222-f4ee-4a59-a270-3f49144fbe1c-kube-api-access-nn8fp\") pod \"d5755222-f4ee-4a59-a270-3f49144fbe1c\" (UID: \"d5755222-f4ee-4a59-a270-3f49144fbe1c\") " Sep 08 04:30:37.555915 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.555899 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5755222-f4ee-4a59-a270-3f49144fbe1c-plugins-config-volume\") pod \"d5755222-f4ee-4a59-a270-3f49144fbe1c\" (UID: \"d5755222-f4ee-4a59-a270-3f49144fbe1c\") " Sep 08 04:30:37.556192 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.556172 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5nbc\" (UniqueName: \"kubernetes.io/projected/abf4d06a-d520-498e-a748-06daf5d53fdb-kube-api-access-z5nbc\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:30:37.556261 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.556203 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf4d06a-d520-498e-a748-06daf5d53fdb-plugins-config-volume\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:30:37.556261 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.556175 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5755222-f4ee-4a59-a270-3f49144fbe1c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d5755222-f4ee-4a59-a270-3f49144fbe1c" (UID: "d5755222-f4ee-4a59-a270-3f49144fbe1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:30:37.557624 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.557609 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5755222-f4ee-4a59-a270-3f49144fbe1c-kube-api-access-nn8fp" (OuterVolumeSpecName: "kube-api-access-nn8fp") pod "d5755222-f4ee-4a59-a270-3f49144fbe1c" (UID: "d5755222-f4ee-4a59-a270-3f49144fbe1c"). InnerVolumeSpecName "kube-api-access-nn8fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:30:37.657487 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.657432 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5755222-f4ee-4a59-a270-3f49144fbe1c-plugins-config-volume\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:30:37.657487 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.657454 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn8fp\" (UniqueName: \"kubernetes.io/projected/d5755222-f4ee-4a59-a270-3f49144fbe1c-kube-api-access-nn8fp\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:30:37.704431 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.704404 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:30:37.827706 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.827678 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8"] Sep 08 04:30:37.833301 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:37.833270 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-431421-658c9c8498-54wr8"] Sep 08 04:30:38.567787 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:38.567753 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abf4d06a-d520-498e-a748-06daf5d53fdb" path="/var/lib/kubelet/pods/abf4d06a-d520-498e-a748-06daf5d53fdb/volumes" Sep 08 04:30:38.568146 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:38.568097 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5755222-f4ee-4a59-a270-3f49144fbe1c" path="/var/lib/kubelet/pods/d5755222-f4ee-4a59-a270-3f49144fbe1c/volumes" Sep 08 04:30:38.568392 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:38.568380 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa2217f9-c043-44f9-bf4e-3130deae06ae" path="/var/lib/kubelet/pods/fa2217f9-c043-44f9-bf4e-3130deae06ae/volumes" Sep 08 04:30:43.985371 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:43.985342 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:30:44.087900 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:44.087870 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:30:44.271334 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:44.271280 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:30:47.774711 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:47.774677 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-520856-4klqd"] Sep 08 04:30:47.775134 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:47.775077 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa2217f9-c043-44f9-bf4e-3130deae06ae" containerName="revoke-keys" Sep 08 04:30:47.775134 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:47.775092 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa2217f9-c043-44f9-bf4e-3130deae06ae" containerName="revoke-keys" Sep 08 04:30:47.775134 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:47.775105 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5755222-f4ee-4a59-a270-3f49144fbe1c" containerName="payload-pre-processing" Sep 08 04:30:47.775134 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:47.775111 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5755222-f4ee-4a59-a270-3f49144fbe1c" containerName="payload-pre-processing" Sep 08 04:30:47.775134 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:47.775129 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abf4d06a-d520-498e-a748-06daf5d53fdb" containerName="payload-processing" Sep 08 04:30:47.775134 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:47.775134 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="abf4d06a-d520-498e-a748-06daf5d53fdb" containerName="payload-processing" Sep 08 04:30:47.775323 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:47.775198 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5755222-f4ee-4a59-a270-3f49144fbe1c" containerName="payload-pre-processing" Sep 08 04:30:47.775323 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:47.775206 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="abf4d06a-d520-498e-a748-06daf5d53fdb" containerName="payload-processing" Sep 08 04:30:47.775323 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:47.775214 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa2217f9-c043-44f9-bf4e-3130deae06ae" containerName="revoke-keys" Sep 08 04:30:47.779651 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:47.779633 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-520856-4klqd" Sep 08 04:30:47.782546 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:47.782521 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 04:30:47.782661 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:47.782543 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7r29w\"" Sep 08 04:30:47.788162 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:47.788138 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-520856-4klqd"] Sep 08 04:30:47.847219 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:47.847187 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4994acfd-4570-4583-8c26-0e0eef446c01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-520856-4klqd\" (UID: \"4994acfd-4570-4583-8c26-0e0eef446c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-520856-4klqd" Sep 08 04:30:47.947889 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:47.947853 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4994acfd-4570-4583-8c26-0e0eef446c01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-520856-4klqd\" (UID: \"4994acfd-4570-4583-8c26-0e0eef446c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-520856-4klqd" Sep 08 04:30:47.948466 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:47.948448 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4994acfd-4570-4583-8c26-0e0eef446c01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-520856-4klqd\" (UID: \"4994acfd-4570-4583-8c26-0e0eef446c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-520856-4klqd" Sep 08 04:30:48.052673 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:48.052589 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:30:48.091557 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:48.091527 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-520856-4klqd" Sep 08 04:30:48.372621 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:48.372421 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-520856-4klqd"] Sep 08 04:30:48.374978 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:30:48.374938 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4994acfd_4570_4583_8c26_0e0eef446c01.slice/crio-49e1765bbea4b4371435923cb844af50ac40e1993871e9baccdb3621cc60e9ea WatchSource:0}: Error finding container 49e1765bbea4b4371435923cb844af50ac40e1993871e9baccdb3621cc60e9ea: Status 404 returned error can't find the container with id 49e1765bbea4b4371435923cb844af50ac40e1993871e9baccdb3621cc60e9ea Sep 08 04:30:48.582498 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:48.582455 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-520856-4klqd" event={"ID":"4994acfd-4570-4583-8c26-0e0eef446c01","Type":"ContainerStarted","Data":"49e1765bbea4b4371435923cb844af50ac40e1993871e9baccdb3621cc60e9ea"} Sep 08 04:30:49.589732 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:49.589698 2574 generic.go:356] "Generic (PLEG): container finished" podID="4994acfd-4570-4583-8c26-0e0eef446c01" containerID="825a596b4aee121976f38babc414431d38958bd8cc44a84ebebabd2c38b304e1" exitCode=0 Sep 08 04:30:49.590167 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:49.589788 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-520856-4klqd" event={"ID":"4994acfd-4570-4583-8c26-0e0eef446c01","Type":"ContainerDied","Data":"825a596b4aee121976f38babc414431d38958bd8cc44a84ebebabd2c38b304e1"} Sep 08 04:30:50.728720 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:50.728694 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-520856-4klqd" Sep 08 04:30:50.775826 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:50.775797 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4994acfd-4570-4583-8c26-0e0eef446c01-maas-api-service-ca\") pod \"4994acfd-4570-4583-8c26-0e0eef446c01\" (UID: \"4994acfd-4570-4583-8c26-0e0eef446c01\") " Sep 08 04:30:50.776144 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:50.776121 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4994acfd-4570-4583-8c26-0e0eef446c01-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4994acfd-4570-4583-8c26-0e0eef446c01" (UID: "4994acfd-4570-4583-8c26-0e0eef446c01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:30:50.877188 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:50.877114 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4994acfd-4570-4583-8c26-0e0eef446c01-maas-api-service-ca\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:30:51.602116 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:51.602086 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-520856-4klqd" Sep 08 04:30:51.602312 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:51.602127 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-520856-4klqd" event={"ID":"4994acfd-4570-4583-8c26-0e0eef446c01","Type":"ContainerDied","Data":"49e1765bbea4b4371435923cb844af50ac40e1993871e9baccdb3621cc60e9ea"} Sep 08 04:30:51.602312 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:30:51.602178 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49e1765bbea4b4371435923cb844af50ac40e1993871e9baccdb3621cc60e9ea" Sep 08 04:31:03.446648 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:03.446604 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:31:03.475518 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:03.475471 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6"] Sep 08 04:31:03.475795 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:03.475757 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6" podUID="65066aa3-519f-436f-ae25-5a20d1140232" containerName="payload-processing" containerID="cri-o://e7c88145e058a5dda7331695c8a0823ad45a0f19df44a017520d02c53e9742e3" gracePeriod=30 Sep 08 04:31:03.562326 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:03.562220 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:31:03.678297 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:03.678265 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-520856-4klqd"] Sep 08 04:31:03.680491 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:03.680368 2574 generic.go:356] "Generic (PLEG): container finished" podID="65066aa3-519f-436f-ae25-5a20d1140232" containerID="e7c88145e058a5dda7331695c8a0823ad45a0f19df44a017520d02c53e9742e3" exitCode=0 Sep 08 04:31:03.680658 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:03.680462 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6" event={"ID":"65066aa3-519f-436f-ae25-5a20d1140232","Type":"ContainerDied","Data":"e7c88145e058a5dda7331695c8a0823ad45a0f19df44a017520d02c53e9742e3"} Sep 08 04:31:03.684035 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:03.684009 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-520856-4klqd"] Sep 08 04:31:03.733901 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:03.733879 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6" Sep 08 04:31:03.766962 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:03.766848 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:31:03.801771 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:03.801736 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65066aa3-519f-436f-ae25-5a20d1140232-plugins-config-volume\") pod \"65066aa3-519f-436f-ae25-5a20d1140232\" (UID: \"65066aa3-519f-436f-ae25-5a20d1140232\") " Sep 08 04:31:03.801958 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:03.801800 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbch7\" (UniqueName: \"kubernetes.io/projected/65066aa3-519f-436f-ae25-5a20d1140232-kube-api-access-lbch7\") pod \"65066aa3-519f-436f-ae25-5a20d1140232\" (UID: \"65066aa3-519f-436f-ae25-5a20d1140232\") " Sep 08 04:31:03.802124 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:03.802099 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65066aa3-519f-436f-ae25-5a20d1140232-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "65066aa3-519f-436f-ae25-5a20d1140232" (UID: "65066aa3-519f-436f-ae25-5a20d1140232"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:31:03.803804 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:03.803783 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65066aa3-519f-436f-ae25-5a20d1140232-kube-api-access-lbch7" (OuterVolumeSpecName: "kube-api-access-lbch7") pod "65066aa3-519f-436f-ae25-5a20d1140232" (UID: "65066aa3-519f-436f-ae25-5a20d1140232"). InnerVolumeSpecName "kube-api-access-lbch7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:31:03.827568 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:03.827457 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:31:03.903120 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:03.903092 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65066aa3-519f-436f-ae25-5a20d1140232-plugins-config-volume\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:31:03.903120 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:03.903118 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbch7\" (UniqueName: \"kubernetes.io/projected/65066aa3-519f-436f-ae25-5a20d1140232-kube-api-access-lbch7\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:31:04.071266 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:04.071240 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:31:04.570056 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:04.570030 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4994acfd-4570-4583-8c26-0e0eef446c01" path="/var/lib/kubelet/pods/4994acfd-4570-4583-8c26-0e0eef446c01/volumes" Sep 08 04:31:04.687684 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:04.687656 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6" Sep 08 04:31:04.687843 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:04.687659 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6" event={"ID":"65066aa3-519f-436f-ae25-5a20d1140232","Type":"ContainerDied","Data":"f04b8df105a95168f99401daba9627bd0e30abac5bd383f7aac5e1ff55240a5c"} Sep 08 04:31:04.687843 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:04.687795 2574 scope.go:117] "RemoveContainer" containerID="e7c88145e058a5dda7331695c8a0823ad45a0f19df44a017520d02c53e9742e3" Sep 08 04:31:04.706084 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:04.706057 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6"] Sep 08 04:31:04.710081 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:04.710057 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-520856-7dc4f47677-nh8h6"] Sep 08 04:31:06.568457 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:06.568417 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65066aa3-519f-436f-ae25-5a20d1140232" path="/var/lib/kubelet/pods/65066aa3-519f-436f-ae25-5a20d1140232/volumes" Sep 08 04:31:09.136452 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:09.136417 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:31:11.577468 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:11.577433 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:31:41.891717 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:41.891679 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:31:42.192188 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:42.192103 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:31:47.965756 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:47.965721 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:31:48.099382 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:48.099342 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:31:48.221994 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:48.221914 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk"] Sep 08 04:31:48.222194 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:48.222166 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" podUID="8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55" containerName="istio-proxy" containerID="cri-o://6b6673f7ca737f1f9eb562295445c8b63d2793f8af4804bd0488b16376574f4d" gracePeriod=30 Sep 08 04:31:48.254039 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:48.254012 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:31:48.611389 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:48.611354 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c84ae-skqgv"] Sep 08 04:31:48.611816 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:48.611797 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65066aa3-519f-436f-ae25-5a20d1140232" containerName="payload-processing" Sep 08 04:31:48.611925 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:48.611818 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="65066aa3-519f-436f-ae25-5a20d1140232" containerName="payload-processing" Sep 08 04:31:48.611925 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:48.611831 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4994acfd-4570-4583-8c26-0e0eef446c01" containerName="revoke-keys" Sep 08 04:31:48.611925 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:48.611839 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4994acfd-4570-4583-8c26-0e0eef446c01" containerName="revoke-keys" Sep 08 04:31:48.612077 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:48.611927 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4994acfd-4570-4583-8c26-0e0eef446c01" containerName="revoke-keys" Sep 08 04:31:48.612077 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:48.611939 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="65066aa3-519f-436f-ae25-5a20d1140232" containerName="payload-processing" Sep 08 04:31:48.615227 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:48.615208 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c84ae-skqgv" Sep 08 04:31:48.619146 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:48.619127 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7r29w\"" Sep 08 04:31:48.619240 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:48.619145 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 04:31:48.632784 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:48.632762 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c84ae-skqgv"] Sep 08 04:31:48.676748 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:48.676721 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/247e8db0-c1bd-4fa6-ac02-105339ab529e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6c84ae-skqgv\" (UID: \"247e8db0-c1bd-4fa6-ac02-105339ab529e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c84ae-skqgv" Sep 08 04:31:48.778102 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:48.778078 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/247e8db0-c1bd-4fa6-ac02-105339ab529e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6c84ae-skqgv\" (UID: \"247e8db0-c1bd-4fa6-ac02-105339ab529e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c84ae-skqgv" Sep 08 04:31:48.778678 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:48.778660 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/247e8db0-c1bd-4fa6-ac02-105339ab529e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6c84ae-skqgv\" (UID: \"247e8db0-c1bd-4fa6-ac02-105339ab529e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c84ae-skqgv" Sep 08 04:31:48.886415 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:48.886313 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:31:48.925983 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:48.925962 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c84ae-skqgv" Sep 08 04:31:49.205154 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:49.205131 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c84ae-skqgv"] Sep 08 04:31:49.206684 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:31:49.206656 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247e8db0_c1bd_4fa6_ac02_105339ab529e.slice/crio-c33ce5e23bd143d8e0963eecb2e7a3a87a81a47f2409b7e860fac531e4ab5a3f WatchSource:0}: Error finding container c33ce5e23bd143d8e0963eecb2e7a3a87a81a47f2409b7e860fac531e4ab5a3f: Status 404 returned error can't find the container with id c33ce5e23bd143d8e0963eecb2e7a3a87a81a47f2409b7e860fac531e4ab5a3f Sep 08 04:31:49.984406 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:49.984379 2574 generic.go:356] "Generic (PLEG): container finished" podID="247e8db0-c1bd-4fa6-ac02-105339ab529e" containerID="ab5919b4dfc5a1574065f1058ae118986f08ed7a6a2340bace1a79bc244d6f20" exitCode=0 Sep 08 04:31:49.984543 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:49.984459 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c84ae-skqgv" event={"ID":"247e8db0-c1bd-4fa6-ac02-105339ab529e","Type":"ContainerDied","Data":"ab5919b4dfc5a1574065f1058ae118986f08ed7a6a2340bace1a79bc244d6f20"} Sep 08 04:31:49.984543 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:49.984512 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c84ae-skqgv" event={"ID":"247e8db0-c1bd-4fa6-ac02-105339ab529e","Type":"ContainerStarted","Data":"c33ce5e23bd143d8e0963eecb2e7a3a87a81a47f2409b7e860fac531e4ab5a3f"} Sep 08 04:31:51.116823 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:51.116799 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c84ae-skqgv" Sep 08 04:31:51.196814 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:51.196787 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/247e8db0-c1bd-4fa6-ac02-105339ab529e-maas-api-service-ca\") pod \"247e8db0-c1bd-4fa6-ac02-105339ab529e\" (UID: \"247e8db0-c1bd-4fa6-ac02-105339ab529e\") " Sep 08 04:31:51.197109 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:51.197087 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247e8db0-c1bd-4fa6-ac02-105339ab529e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "247e8db0-c1bd-4fa6-ac02-105339ab529e" (UID: "247e8db0-c1bd-4fa6-ac02-105339ab529e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:31:51.297613 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:51.297587 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/247e8db0-c1bd-4fa6-ac02-105339ab529e-maas-api-service-ca\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:31:51.997916 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:51.997893 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c84ae-skqgv" Sep 08 04:31:51.998060 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:51.997918 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c84ae-skqgv" event={"ID":"247e8db0-c1bd-4fa6-ac02-105339ab529e","Type":"ContainerDied","Data":"c33ce5e23bd143d8e0963eecb2e7a3a87a81a47f2409b7e860fac531e4ab5a3f"} Sep 08 04:31:51.998060 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:51.997968 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c33ce5e23bd143d8e0963eecb2e7a3a87a81a47f2409b7e860fac531e4ab5a3f" Sep 08 04:31:53.471741 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.471716 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:31:53.616178 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.616151 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrjls\" (UniqueName: \"kubernetes.io/projected/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-kube-api-access-lrjls\") pod \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " Sep 08 04:31:53.616309 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.616207 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istiod-ca-cert\") pod \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " Sep 08 04:31:53.616309 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.616253 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-workload-socket\") pod \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " Sep 08 04:31:53.616309 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.616268 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istio-token\") pod \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " Sep 08 04:31:53.616309 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.616297 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istio-envoy\") pod \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " Sep 08 04:31:53.616534 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.616328 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-workload-certs\") pod \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " Sep 08 04:31:53.616534 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.616354 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istio-data\") pod \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " Sep 08 04:31:53.616534 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.616394 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istio-podinfo\") pod \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " Sep 08 04:31:53.616534 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.616466 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-credential-socket\") pod \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\" (UID: \"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55\") " Sep 08 04:31:53.616759 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.616617 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55" (UID: "8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:31:53.616823 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.616763 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istio-data" (OuterVolumeSpecName: "istio-data") pod "8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55" (UID: "8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:31:53.616823 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.616780 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-workload-socket\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:31:53.616823 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.616722 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55" (UID: "8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:31:53.616979 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.616817 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55" (UID: "8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:31:53.616979 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.616931 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55" (UID: "8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:31:53.618560 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.618531 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-kube-api-access-lrjls" (OuterVolumeSpecName: "kube-api-access-lrjls") pod "8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55" (UID: "8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55"). InnerVolumeSpecName "kube-api-access-lrjls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:31:53.618560 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.618543 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55" (UID: "8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:31:53.618797 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.618779 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istio-token" (OuterVolumeSpecName: "istio-token") pod "8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55" (UID: "8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:31:53.618894 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.618875 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55" (UID: "8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 04:31:53.717178 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.717151 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istio-podinfo\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:31:53.717178 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.717175 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-credential-socket\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:31:53.717338 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.717189 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrjls\" (UniqueName: \"kubernetes.io/projected/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-kube-api-access-lrjls\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:31:53.717338 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.717202 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istiod-ca-cert\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:31:53.717338 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.717221 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istio-token\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:31:53.717338 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.717233 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istio-envoy\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:31:53.717338 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.717245 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-workload-certs\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:31:53.717338 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:53.717256 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55-istio-data\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:31:54.012697 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:54.012662 2574 generic.go:356] "Generic (PLEG): container finished" podID="8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55" containerID="6b6673f7ca737f1f9eb562295445c8b63d2793f8af4804bd0488b16376574f4d" exitCode=0 Sep 08 04:31:54.012807 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:54.012722 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" Sep 08 04:31:54.012889 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:54.012726 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" event={"ID":"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55","Type":"ContainerDied","Data":"6b6673f7ca737f1f9eb562295445c8b63d2793f8af4804bd0488b16376574f4d"} Sep 08 04:31:54.012889 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:54.012861 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk" event={"ID":"8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55","Type":"ContainerDied","Data":"d666ab1db82f219998e4ab1741d0f12119caf1ce931c4e30a7e549d0153a4bec"} Sep 08 04:31:54.012993 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:54.012901 2574 scope.go:117] "RemoveContainer" containerID="6b6673f7ca737f1f9eb562295445c8b63d2793f8af4804bd0488b16376574f4d" Sep 08 04:31:54.023075 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:54.023053 2574 scope.go:117] "RemoveContainer" containerID="ac4102a274fd5e081b0a4efe72f77af4bc1fe00c19e6dfdd5c934c1147473180" Sep 08 04:31:54.032316 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:54.032298 2574 scope.go:117] "RemoveContainer" containerID="6b6673f7ca737f1f9eb562295445c8b63d2793f8af4804bd0488b16376574f4d" Sep 08 04:31:54.032614 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:31:54.032596 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b6673f7ca737f1f9eb562295445c8b63d2793f8af4804bd0488b16376574f4d\": container with ID starting with 6b6673f7ca737f1f9eb562295445c8b63d2793f8af4804bd0488b16376574f4d not found: ID does not exist" containerID="6b6673f7ca737f1f9eb562295445c8b63d2793f8af4804bd0488b16376574f4d" Sep 08 04:31:54.032672 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:54.032623 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b6673f7ca737f1f9eb562295445c8b63d2793f8af4804bd0488b16376574f4d"} err="failed to get container status \"6b6673f7ca737f1f9eb562295445c8b63d2793f8af4804bd0488b16376574f4d\": rpc error: code = NotFound desc = could not find container \"6b6673f7ca737f1f9eb562295445c8b63d2793f8af4804bd0488b16376574f4d\": container with ID starting with 6b6673f7ca737f1f9eb562295445c8b63d2793f8af4804bd0488b16376574f4d not found: ID does not exist" Sep 08 04:31:54.032672 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:54.032641 2574 scope.go:117] "RemoveContainer" containerID="ac4102a274fd5e081b0a4efe72f77af4bc1fe00c19e6dfdd5c934c1147473180" Sep 08 04:31:54.032930 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:31:54.032909 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac4102a274fd5e081b0a4efe72f77af4bc1fe00c19e6dfdd5c934c1147473180\": container with ID starting with ac4102a274fd5e081b0a4efe72f77af4bc1fe00c19e6dfdd5c934c1147473180 not found: ID does not exist" containerID="ac4102a274fd5e081b0a4efe72f77af4bc1fe00c19e6dfdd5c934c1147473180" Sep 08 04:31:54.032978 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:54.032939 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac4102a274fd5e081b0a4efe72f77af4bc1fe00c19e6dfdd5c934c1147473180"} err="failed to get container status \"ac4102a274fd5e081b0a4efe72f77af4bc1fe00c19e6dfdd5c934c1147473180\": rpc error: code = NotFound desc = could not find container \"ac4102a274fd5e081b0a4efe72f77af4bc1fe00c19e6dfdd5c934c1147473180\": container with ID starting with ac4102a274fd5e081b0a4efe72f77af4bc1fe00c19e6dfdd5c934c1147473180 not found: ID does not exist" Sep 08 04:31:54.037065 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:54.037036 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk"] Sep 08 04:31:54.041538 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:54.041498 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-520856-openshift-default-6557c5d99d-g4nqk"] Sep 08 04:31:54.568267 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:54.568237 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55" path="/var/lib/kubelet/pods/8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55/volumes" Sep 08 04:31:59.220780 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:59.220748 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:31:59.225622 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:59.225597 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6c84ae-29814030-m8vnz"] Sep 08 04:31:59.233259 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:59.233232 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6c84ae-29814030-m8vnz"] Sep 08 04:31:59.375470 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:59.375435 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:31:59.478303 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:59.478228 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:31:59.567066 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:59.567038 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c84ae-skqgv"] Sep 08 04:31:59.569988 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:59.569961 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6c84ae-skqgv"] Sep 08 04:31:59.573730 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:59.573698 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:31:59.695309 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:59.695275 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:31:59.967630 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:31:59.967596 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:32:00.567885 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:00.567852 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="247e8db0-c1bd-4fa6-ac02-105339ab529e" path="/var/lib/kubelet/pods/247e8db0-c1bd-4fa6-ac02-105339ab529e/volumes" Sep 08 04:32:00.568256 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:00.568138 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44e4ad2f-2781-469f-ab11-211451e914b5" path="/var/lib/kubelet/pods/44e4ad2f-2781-469f-ab11-211451e914b5/volumes" Sep 08 04:32:05.743481 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:05.743446 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:32:05.807366 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:05.807333 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:32:06.009099 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:06.009065 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg"] Sep 08 04:32:06.009345 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:06.009319 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" podUID="3427054f-6ed2-482b-817a-8f6859f7c383" containerName="istio-proxy" containerID="cri-o://247c845120c619b5bd1e8692569b2aee65cd018f1406a01c6f9137ffaa506b82" gracePeriod=30 Sep 08 04:32:06.035617 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:06.035588 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:32:06.483794 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:06.483705 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:32:11.262437 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.262411 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:32:11.365986 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.365925 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3427054f-6ed2-482b-817a-8f6859f7c383-istiod-ca-cert\") pod \"3427054f-6ed2-482b-817a-8f6859f7c383\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " Sep 08 04:32:11.365986 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.365981 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-istio-envoy\") pod \"3427054f-6ed2-482b-817a-8f6859f7c383\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " Sep 08 04:32:11.366175 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.366007 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6l4s\" (UniqueName: \"kubernetes.io/projected/3427054f-6ed2-482b-817a-8f6859f7c383-kube-api-access-s6l4s\") pod \"3427054f-6ed2-482b-817a-8f6859f7c383\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " Sep 08 04:32:11.366175 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.366025 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-workload-certs\") pod \"3427054f-6ed2-482b-817a-8f6859f7c383\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " Sep 08 04:32:11.366175 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.366050 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3427054f-6ed2-482b-817a-8f6859f7c383-istio-token\") pod \"3427054f-6ed2-482b-817a-8f6859f7c383\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " Sep 08 04:32:11.366175 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.366074 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-credential-socket\") pod \"3427054f-6ed2-482b-817a-8f6859f7c383\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " Sep 08 04:32:11.366175 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.366126 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3427054f-6ed2-482b-817a-8f6859f7c383-istio-podinfo\") pod \"3427054f-6ed2-482b-817a-8f6859f7c383\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " Sep 08 04:32:11.366470 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.366384 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3427054f-6ed2-482b-817a-8f6859f7c383" (UID: "3427054f-6ed2-482b-817a-8f6859f7c383"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:32:11.366470 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.366401 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-istio-data\") pod \"3427054f-6ed2-482b-817a-8f6859f7c383\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " Sep 08 04:32:11.366470 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.366415 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3427054f-6ed2-482b-817a-8f6859f7c383-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3427054f-6ed2-482b-817a-8f6859f7c383" (UID: "3427054f-6ed2-482b-817a-8f6859f7c383"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:32:11.366626 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.366479 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-workload-socket\") pod \"3427054f-6ed2-482b-817a-8f6859f7c383\" (UID: \"3427054f-6ed2-482b-817a-8f6859f7c383\") " Sep 08 04:32:11.366689 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.366656 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-istio-data" (OuterVolumeSpecName: "istio-data") pod "3427054f-6ed2-482b-817a-8f6859f7c383" (UID: "3427054f-6ed2-482b-817a-8f6859f7c383"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:32:11.366858 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.366765 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3427054f-6ed2-482b-817a-8f6859f7c383" (UID: "3427054f-6ed2-482b-817a-8f6859f7c383"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:32:11.366858 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.366789 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3427054f-6ed2-482b-817a-8f6859f7c383" (UID: "3427054f-6ed2-482b-817a-8f6859f7c383"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:32:11.367038 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.366921 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-istio-data\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:32:11.367038 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.366938 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-workload-socket\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:32:11.367038 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.366952 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3427054f-6ed2-482b-817a-8f6859f7c383-istiod-ca-cert\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:32:11.367038 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.366965 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-workload-certs\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:32:11.367038 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.366978 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-credential-socket\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:32:11.368270 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.368245 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3427054f-6ed2-482b-817a-8f6859f7c383" (UID: "3427054f-6ed2-482b-817a-8f6859f7c383"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:32:11.368362 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.368290 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3427054f-6ed2-482b-817a-8f6859f7c383-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3427054f-6ed2-482b-817a-8f6859f7c383" (UID: "3427054f-6ed2-482b-817a-8f6859f7c383"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 04:32:11.368362 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.368291 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3427054f-6ed2-482b-817a-8f6859f7c383-istio-token" (OuterVolumeSpecName: "istio-token") pod "3427054f-6ed2-482b-817a-8f6859f7c383" (UID: "3427054f-6ed2-482b-817a-8f6859f7c383"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:32:11.368448 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.368383 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3427054f-6ed2-482b-817a-8f6859f7c383-kube-api-access-s6l4s" (OuterVolumeSpecName: "kube-api-access-s6l4s") pod "3427054f-6ed2-482b-817a-8f6859f7c383" (UID: "3427054f-6ed2-482b-817a-8f6859f7c383"). InnerVolumeSpecName "kube-api-access-s6l4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:32:11.467626 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.467603 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6l4s\" (UniqueName: \"kubernetes.io/projected/3427054f-6ed2-482b-817a-8f6859f7c383-kube-api-access-s6l4s\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:32:11.467626 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.467623 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3427054f-6ed2-482b-817a-8f6859f7c383-istio-token\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:32:11.467729 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.467632 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3427054f-6ed2-482b-817a-8f6859f7c383-istio-podinfo\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:32:11.467729 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:11.467640 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3427054f-6ed2-482b-817a-8f6859f7c383-istio-envoy\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:32:12.127280 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:12.127243 2574 generic.go:356] "Generic (PLEG): container finished" podID="3427054f-6ed2-482b-817a-8f6859f7c383" containerID="247c845120c619b5bd1e8692569b2aee65cd018f1406a01c6f9137ffaa506b82" exitCode=0 Sep 08 04:32:12.127467 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:12.127296 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" Sep 08 04:32:12.127467 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:12.127319 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" event={"ID":"3427054f-6ed2-482b-817a-8f6859f7c383","Type":"ContainerDied","Data":"247c845120c619b5bd1e8692569b2aee65cd018f1406a01c6f9137ffaa506b82"} Sep 08 04:32:12.127467 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:12.127392 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg" event={"ID":"3427054f-6ed2-482b-817a-8f6859f7c383","Type":"ContainerDied","Data":"889766d55d29ce45092b7adcf72eb15de85ac10c9dc2f7b574f586fe8b21b83b"} Sep 08 04:32:12.127467 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:12.127415 2574 scope.go:117] "RemoveContainer" containerID="247c845120c619b5bd1e8692569b2aee65cd018f1406a01c6f9137ffaa506b82" Sep 08 04:32:12.137851 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:12.137836 2574 scope.go:117] "RemoveContainer" containerID="c7b38113168b8384e754f9d8c606b8957e41c3e3ec3b23a922a7ebeac644443f" Sep 08 04:32:12.146166 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:12.146151 2574 scope.go:117] "RemoveContainer" containerID="247c845120c619b5bd1e8692569b2aee65cd018f1406a01c6f9137ffaa506b82" Sep 08 04:32:12.146433 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:32:12.146416 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"247c845120c619b5bd1e8692569b2aee65cd018f1406a01c6f9137ffaa506b82\": container with ID starting with 247c845120c619b5bd1e8692569b2aee65cd018f1406a01c6f9137ffaa506b82 not found: ID does not exist" containerID="247c845120c619b5bd1e8692569b2aee65cd018f1406a01c6f9137ffaa506b82" Sep 08 04:32:12.146472 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:12.146441 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"247c845120c619b5bd1e8692569b2aee65cd018f1406a01c6f9137ffaa506b82"} err="failed to get container status \"247c845120c619b5bd1e8692569b2aee65cd018f1406a01c6f9137ffaa506b82\": rpc error: code = NotFound desc = could not find container \"247c845120c619b5bd1e8692569b2aee65cd018f1406a01c6f9137ffaa506b82\": container with ID starting with 247c845120c619b5bd1e8692569b2aee65cd018f1406a01c6f9137ffaa506b82 not found: ID does not exist" Sep 08 04:32:12.146472 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:12.146456 2574 scope.go:117] "RemoveContainer" containerID="c7b38113168b8384e754f9d8c606b8957e41c3e3ec3b23a922a7ebeac644443f" Sep 08 04:32:12.146679 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:32:12.146661 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7b38113168b8384e754f9d8c606b8957e41c3e3ec3b23a922a7ebeac644443f\": container with ID starting with c7b38113168b8384e754f9d8c606b8957e41c3e3ec3b23a922a7ebeac644443f not found: ID does not exist" containerID="c7b38113168b8384e754f9d8c606b8957e41c3e3ec3b23a922a7ebeac644443f" Sep 08 04:32:12.146719 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:12.146683 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7b38113168b8384e754f9d8c606b8957e41c3e3ec3b23a922a7ebeac644443f"} err="failed to get container status \"c7b38113168b8384e754f9d8c606b8957e41c3e3ec3b23a922a7ebeac644443f\": rpc error: code = NotFound desc = could not find container \"c7b38113168b8384e754f9d8c606b8957e41c3e3ec3b23a922a7ebeac644443f\": container with ID starting with c7b38113168b8384e754f9d8c606b8957e41c3e3ec3b23a922a7ebeac644443f not found: ID does not exist" Sep 08 04:32:12.152117 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:12.152093 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg"] Sep 08 04:32:12.156574 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:12.156551 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-6c84ae-openshift-default-7b65c76854-c9xtg"] Sep 08 04:32:12.568321 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:12.568284 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3427054f-6ed2-482b-817a-8f6859f7c383" path="/var/lib/kubelet/pods/3427054f-6ed2-482b-817a-8f6859f7c383/volumes" Sep 08 04:32:16.915851 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:16.915814 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:32:16.918873 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:16.918848 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2a8bb-29814030-f8gcd"] Sep 08 04:32:16.922057 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:16.922029 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2a8bb-29814030-f8gcd"] Sep 08 04:32:16.939950 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:16.939921 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8"] Sep 08 04:32:16.940206 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:16.940181 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8" podUID="1eb7575b-685f-48e8-8616-b52b6722357f" containerName="payload-processing" containerID="cri-o://61218c6991f09d802fbcad38744d56e24487b28d3c30a8758e336785a975b254" gracePeriod=30 Sep 08 04:32:17.016662 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:17.016633 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:32:17.112852 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:17.112825 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:32:17.161013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:17.160982 2574 generic.go:356] "Generic (PLEG): container finished" podID="1eb7575b-685f-48e8-8616-b52b6722357f" containerID="61218c6991f09d802fbcad38744d56e24487b28d3c30a8758e336785a975b254" exitCode=0 Sep 08 04:32:17.161162 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:17.161053 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8" event={"ID":"1eb7575b-685f-48e8-8616-b52b6722357f","Type":"ContainerDied","Data":"61218c6991f09d802fbcad38744d56e24487b28d3c30a8758e336785a975b254"} Sep 08 04:32:17.191885 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:17.191859 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8" Sep 08 04:32:17.221968 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:17.221931 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:32:17.313578 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:17.313546 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fngpz\" (UniqueName: \"kubernetes.io/projected/1eb7575b-685f-48e8-8616-b52b6722357f-kube-api-access-fngpz\") pod \"1eb7575b-685f-48e8-8616-b52b6722357f\" (UID: \"1eb7575b-685f-48e8-8616-b52b6722357f\") " Sep 08 04:32:17.313720 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:17.313602 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb7575b-685f-48e8-8616-b52b6722357f-plugins-config-volume\") pod \"1eb7575b-685f-48e8-8616-b52b6722357f\" (UID: \"1eb7575b-685f-48e8-8616-b52b6722357f\") " Sep 08 04:32:17.314012 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:17.313990 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb7575b-685f-48e8-8616-b52b6722357f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1eb7575b-685f-48e8-8616-b52b6722357f" (UID: "1eb7575b-685f-48e8-8616-b52b6722357f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:32:17.315791 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:17.315771 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb7575b-685f-48e8-8616-b52b6722357f-kube-api-access-fngpz" (OuterVolumeSpecName: "kube-api-access-fngpz") pod "1eb7575b-685f-48e8-8616-b52b6722357f" (UID: "1eb7575b-685f-48e8-8616-b52b6722357f"). InnerVolumeSpecName "kube-api-access-fngpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:32:17.414264 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:17.414240 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fngpz\" (UniqueName: \"kubernetes.io/projected/1eb7575b-685f-48e8-8616-b52b6722357f-kube-api-access-fngpz\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:32:17.414264 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:17.414263 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb7575b-685f-48e8-8616-b52b6722357f-plugins-config-volume\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:32:17.588010 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:17.587973 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:32:18.169045 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:18.169021 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8" Sep 08 04:32:18.169396 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:18.169069 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8" event={"ID":"1eb7575b-685f-48e8-8616-b52b6722357f","Type":"ContainerDied","Data":"941b882ffb152309148fe20324b80936c08e8af9b5672ab9d11d5f08b90657cc"} Sep 08 04:32:18.169396 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:18.169131 2574 scope.go:117] "RemoveContainer" containerID="61218c6991f09d802fbcad38744d56e24487b28d3c30a8758e336785a975b254" Sep 08 04:32:18.196523 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:18.196259 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8"] Sep 08 04:32:18.203847 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:18.203622 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c2a8bb-5955b98cbc-rw4x8"] Sep 08 04:32:18.567872 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:18.567831 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1eb7575b-685f-48e8-8616-b52b6722357f" path="/var/lib/kubelet/pods/1eb7575b-685f-48e8-8616-b52b6722357f/volumes" Sep 08 04:32:18.568197 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:18.568174 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6207719-c7aa-45a8-a2cb-8eb7ed081c76" path="/var/lib/kubelet/pods/d6207719-c7aa-45a8-a2cb-8eb7ed081c76/volumes" Sep 08 04:32:23.644398 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:23.644344 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:32:23.706460 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:23.706428 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:32:23.881771 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:23.881737 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt"] Sep 08 04:32:23.881960 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:23.881937 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" podUID="fc67f27e-765a-425a-9b20-d7d05f43301c" containerName="istio-proxy" containerID="cri-o://c7107cc712ca3f7e914a8470246491866ec982f8943fc4e004a5673cddba420b" gracePeriod=30 Sep 08 04:32:23.905603 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:23.905541 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:32:24.471469 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:24.471435 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:32:26.965593 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:26.965548 2574 scope.go:117] "RemoveContainer" containerID="7e085045221a7591bd16f75481cb302f7829eed22ee218945b4f810a8a8e13d9" Sep 08 04:32:26.975323 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:26.975298 2574 scope.go:117] "RemoveContainer" containerID="6f48382327c762d648a059d299970d96a78ccfc084b88425f5bda621c09ff63c" Sep 08 04:32:26.983664 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:26.983641 2574 scope.go:117] "RemoveContainer" containerID="0813f790b56152b1735561f58c070d6aa8304c6daa2d56e8ef29fa2906cd46bf" Sep 08 04:32:29.134541 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.134500 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:32:29.203621 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.203562 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-workload-certs\") pod \"fc67f27e-765a-425a-9b20-d7d05f43301c\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " Sep 08 04:32:29.203621 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.203598 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc67f27e-765a-425a-9b20-d7d05f43301c-istio-token\") pod \"fc67f27e-765a-425a-9b20-d7d05f43301c\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " Sep 08 04:32:29.203798 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.203626 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-workload-socket\") pod \"fc67f27e-765a-425a-9b20-d7d05f43301c\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " Sep 08 04:32:29.203798 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.203660 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-istio-data\") pod \"fc67f27e-765a-425a-9b20-d7d05f43301c\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " Sep 08 04:32:29.203798 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.203696 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-istio-envoy\") pod \"fc67f27e-765a-425a-9b20-d7d05f43301c\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " Sep 08 04:32:29.203798 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.203732 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-credential-socket\") pod \"fc67f27e-765a-425a-9b20-d7d05f43301c\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " Sep 08 04:32:29.203798 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.203783 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc67f27e-765a-425a-9b20-d7d05f43301c-istio-podinfo\") pod \"fc67f27e-765a-425a-9b20-d7d05f43301c\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " Sep 08 04:32:29.204022 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.203810 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljf6g\" (UniqueName: \"kubernetes.io/projected/fc67f27e-765a-425a-9b20-d7d05f43301c-kube-api-access-ljf6g\") pod \"fc67f27e-765a-425a-9b20-d7d05f43301c\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " Sep 08 04:32:29.204022 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.203831 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fc67f27e-765a-425a-9b20-d7d05f43301c" (UID: "fc67f27e-765a-425a-9b20-d7d05f43301c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:32:29.204022 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.203903 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc67f27e-765a-425a-9b20-d7d05f43301c-istiod-ca-cert\") pod \"fc67f27e-765a-425a-9b20-d7d05f43301c\" (UID: \"fc67f27e-765a-425a-9b20-d7d05f43301c\") " Sep 08 04:32:29.204022 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.203941 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fc67f27e-765a-425a-9b20-d7d05f43301c" (UID: "fc67f27e-765a-425a-9b20-d7d05f43301c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:32:29.204478 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.204414 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fc67f27e-765a-425a-9b20-d7d05f43301c" (UID: "fc67f27e-765a-425a-9b20-d7d05f43301c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:32:29.204878 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.203922 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-istio-data" (OuterVolumeSpecName: "istio-data") pod "fc67f27e-765a-425a-9b20-d7d05f43301c" (UID: "fc67f27e-765a-425a-9b20-d7d05f43301c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:32:29.205239 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.205182 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc67f27e-765a-425a-9b20-d7d05f43301c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fc67f27e-765a-425a-9b20-d7d05f43301c" (UID: "fc67f27e-765a-425a-9b20-d7d05f43301c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:32:29.205767 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.205629 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-credential-socket\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:32:29.205767 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.205657 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc67f27e-765a-425a-9b20-d7d05f43301c-istiod-ca-cert\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:32:29.205767 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.205674 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-workload-certs\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:32:29.205767 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.205696 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-workload-socket\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:32:29.205767 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.205709 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-istio-data\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:32:29.206406 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.206111 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc67f27e-765a-425a-9b20-d7d05f43301c-istio-token" (OuterVolumeSpecName: "istio-token") pod "fc67f27e-765a-425a-9b20-d7d05f43301c" (UID: "fc67f27e-765a-425a-9b20-d7d05f43301c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:32:29.206796 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.206760 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc67f27e-765a-425a-9b20-d7d05f43301c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fc67f27e-765a-425a-9b20-d7d05f43301c" (UID: "fc67f27e-765a-425a-9b20-d7d05f43301c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 04:32:29.212948 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.212918 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fc67f27e-765a-425a-9b20-d7d05f43301c" (UID: "fc67f27e-765a-425a-9b20-d7d05f43301c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 04:32:29.212948 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.212917 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc67f27e-765a-425a-9b20-d7d05f43301c-kube-api-access-ljf6g" (OuterVolumeSpecName: "kube-api-access-ljf6g") pod "fc67f27e-765a-425a-9b20-d7d05f43301c" (UID: "fc67f27e-765a-425a-9b20-d7d05f43301c"). InnerVolumeSpecName "kube-api-access-ljf6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:32:29.239188 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.239165 2574 generic.go:356] "Generic (PLEG): container finished" podID="fc67f27e-765a-425a-9b20-d7d05f43301c" containerID="c7107cc712ca3f7e914a8470246491866ec982f8943fc4e004a5673cddba420b" exitCode=0 Sep 08 04:32:29.239289 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.239212 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" event={"ID":"fc67f27e-765a-425a-9b20-d7d05f43301c","Type":"ContainerDied","Data":"c7107cc712ca3f7e914a8470246491866ec982f8943fc4e004a5673cddba420b"} Sep 08 04:32:29.239289 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.239228 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" Sep 08 04:32:29.239289 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.239275 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt" event={"ID":"fc67f27e-765a-425a-9b20-d7d05f43301c","Type":"ContainerDied","Data":"6fdcfb0f2c697b48441b64e7cda1529378859f6922b27c8650283a10f631732b"} Sep 08 04:32:29.239443 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.239296 2574 scope.go:117] "RemoveContainer" containerID="c7107cc712ca3f7e914a8470246491866ec982f8943fc4e004a5673cddba420b" Sep 08 04:32:29.251771 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.251754 2574 scope.go:117] "RemoveContainer" containerID="f2f7558d659683be33ebb0c84976d6c4088d8d6248c8076b1b9be94e8edfadac" Sep 08 04:32:29.259798 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.259773 2574 scope.go:117] "RemoveContainer" containerID="c7107cc712ca3f7e914a8470246491866ec982f8943fc4e004a5673cddba420b" Sep 08 04:32:29.260046 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:32:29.260030 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7107cc712ca3f7e914a8470246491866ec982f8943fc4e004a5673cddba420b\": container with ID starting with c7107cc712ca3f7e914a8470246491866ec982f8943fc4e004a5673cddba420b not found: ID does not exist" containerID="c7107cc712ca3f7e914a8470246491866ec982f8943fc4e004a5673cddba420b" Sep 08 04:32:29.260108 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.260052 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7107cc712ca3f7e914a8470246491866ec982f8943fc4e004a5673cddba420b"} err="failed to get container status \"c7107cc712ca3f7e914a8470246491866ec982f8943fc4e004a5673cddba420b\": rpc error: code = NotFound desc = could not find container \"c7107cc712ca3f7e914a8470246491866ec982f8943fc4e004a5673cddba420b\": container with ID starting with c7107cc712ca3f7e914a8470246491866ec982f8943fc4e004a5673cddba420b not found: ID does not exist" Sep 08 04:32:29.260108 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.260069 2574 scope.go:117] "RemoveContainer" containerID="f2f7558d659683be33ebb0c84976d6c4088d8d6248c8076b1b9be94e8edfadac" Sep 08 04:32:29.260266 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:32:29.260253 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2f7558d659683be33ebb0c84976d6c4088d8d6248c8076b1b9be94e8edfadac\": container with ID starting with f2f7558d659683be33ebb0c84976d6c4088d8d6248c8076b1b9be94e8edfadac not found: ID does not exist" containerID="f2f7558d659683be33ebb0c84976d6c4088d8d6248c8076b1b9be94e8edfadac" Sep 08 04:32:29.260305 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.260268 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2f7558d659683be33ebb0c84976d6c4088d8d6248c8076b1b9be94e8edfadac"} err="failed to get container status \"f2f7558d659683be33ebb0c84976d6c4088d8d6248c8076b1b9be94e8edfadac\": rpc error: code = NotFound desc = could not find container \"f2f7558d659683be33ebb0c84976d6c4088d8d6248c8076b1b9be94e8edfadac\": container with ID starting with f2f7558d659683be33ebb0c84976d6c4088d8d6248c8076b1b9be94e8edfadac not found: ID does not exist" Sep 08 04:32:29.262020 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.262001 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt"] Sep 08 04:32:29.265749 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.265729 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c2a8bb-openshift-default-b5fc6b6d-95hvt"] Sep 08 04:32:29.306722 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.306699 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc67f27e-765a-425a-9b20-d7d05f43301c-istio-token\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:32:29.306722 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.306718 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc67f27e-765a-425a-9b20-d7d05f43301c-istio-envoy\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:32:29.306722 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.306726 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc67f27e-765a-425a-9b20-d7d05f43301c-istio-podinfo\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:32:29.306866 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:29.306735 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljf6g\" (UniqueName: \"kubernetes.io/projected/fc67f27e-765a-425a-9b20-d7d05f43301c-kube-api-access-ljf6g\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:32:30.568244 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:30.568193 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc67f27e-765a-425a-9b20-d7d05f43301c" path="/var/lib/kubelet/pods/fc67f27e-765a-425a-9b20-d7d05f43301c/volumes" Sep 08 04:32:34.817999 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:34.817965 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:32:34.829617 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:34.829584 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-720789-29814030-k4p6r"] Sep 08 04:32:34.831780 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:34.831755 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-720789-29814030-k4p6r"] Sep 08 04:32:34.866792 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:34.866763 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh"] Sep 08 04:32:34.867364 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:34.867330 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh" podUID="f150eabb-c9b7-4498-8f06-8f9243d8f058" containerName="payload-processing" containerID="cri-o://96da1c398575c819a35899074950b853b021d28972a35d4199d131fca13ee7cb" gracePeriod=30 Sep 08 04:32:34.872412 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:34.872390 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7"] Sep 08 04:32:34.872723 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:34.872701 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7" podUID="83d7d5f8-f5da-406a-a976-388737723147" containerName="payload-pre-processing" containerID="cri-o://6444adb957bbf186fb5e7d6f36dfaf8dfad0d8e8c252c00f156cc6a80d4f8cb6" gracePeriod=30 Sep 08 04:32:34.896246 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:34.896134 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:32:34.969304 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:34.969278 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:32:35.044447 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.042798 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:32:35.111920 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.111895 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:32:35.137211 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.137188 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh" Sep 08 04:32:35.139875 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.139850 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7" Sep 08 04:32:35.245978 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.245945 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f150eabb-c9b7-4498-8f06-8f9243d8f058-plugins-config-volume\") pod \"f150eabb-c9b7-4498-8f06-8f9243d8f058\" (UID: \"f150eabb-c9b7-4498-8f06-8f9243d8f058\") " Sep 08 04:32:35.246140 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.245994 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlcdr\" (UniqueName: \"kubernetes.io/projected/f150eabb-c9b7-4498-8f06-8f9243d8f058-kube-api-access-mlcdr\") pod \"f150eabb-c9b7-4498-8f06-8f9243d8f058\" (UID: \"f150eabb-c9b7-4498-8f06-8f9243d8f058\") " Sep 08 04:32:35.246140 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.246075 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83d7d5f8-f5da-406a-a976-388737723147-plugins-config-volume\") pod \"83d7d5f8-f5da-406a-a976-388737723147\" (UID: \"83d7d5f8-f5da-406a-a976-388737723147\") " Sep 08 04:32:35.246140 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.246117 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfw2r\" (UniqueName: \"kubernetes.io/projected/83d7d5f8-f5da-406a-a976-388737723147-kube-api-access-sfw2r\") pod \"83d7d5f8-f5da-406a-a976-388737723147\" (UID: \"83d7d5f8-f5da-406a-a976-388737723147\") " Sep 08 04:32:35.246476 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.246453 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f150eabb-c9b7-4498-8f06-8f9243d8f058-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f150eabb-c9b7-4498-8f06-8f9243d8f058" (UID: "f150eabb-c9b7-4498-8f06-8f9243d8f058"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:32:35.246607 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.246489 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83d7d5f8-f5da-406a-a976-388737723147-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "83d7d5f8-f5da-406a-a976-388737723147" (UID: "83d7d5f8-f5da-406a-a976-388737723147"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 04:32:35.248049 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.248030 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f150eabb-c9b7-4498-8f06-8f9243d8f058-kube-api-access-mlcdr" (OuterVolumeSpecName: "kube-api-access-mlcdr") pod "f150eabb-c9b7-4498-8f06-8f9243d8f058" (UID: "f150eabb-c9b7-4498-8f06-8f9243d8f058"). InnerVolumeSpecName "kube-api-access-mlcdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:32:35.248151 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.248129 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83d7d5f8-f5da-406a-a976-388737723147-kube-api-access-sfw2r" (OuterVolumeSpecName: "kube-api-access-sfw2r") pod "83d7d5f8-f5da-406a-a976-388737723147" (UID: "83d7d5f8-f5da-406a-a976-388737723147"). InnerVolumeSpecName "kube-api-access-sfw2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 04:32:35.276393 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.276367 2574 generic.go:356] "Generic (PLEG): container finished" podID="83d7d5f8-f5da-406a-a976-388737723147" containerID="6444adb957bbf186fb5e7d6f36dfaf8dfad0d8e8c252c00f156cc6a80d4f8cb6" exitCode=0 Sep 08 04:32:35.276516 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.276409 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7" Sep 08 04:32:35.276516 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.276449 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7" event={"ID":"83d7d5f8-f5da-406a-a976-388737723147","Type":"ContainerDied","Data":"6444adb957bbf186fb5e7d6f36dfaf8dfad0d8e8c252c00f156cc6a80d4f8cb6"} Sep 08 04:32:35.276516 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.276497 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7" event={"ID":"83d7d5f8-f5da-406a-a976-388737723147","Type":"ContainerDied","Data":"02737c3f7ffe307e4be8550a868f011e9128a42e402d0cfda43a5ddfef77ed23"} Sep 08 04:32:35.276697 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.276522 2574 scope.go:117] "RemoveContainer" containerID="6444adb957bbf186fb5e7d6f36dfaf8dfad0d8e8c252c00f156cc6a80d4f8cb6" Sep 08 04:32:35.277828 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.277806 2574 generic.go:356] "Generic (PLEG): container finished" podID="f150eabb-c9b7-4498-8f06-8f9243d8f058" containerID="96da1c398575c819a35899074950b853b021d28972a35d4199d131fca13ee7cb" exitCode=0 Sep 08 04:32:35.277917 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.277842 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh" Sep 08 04:32:35.277917 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.277882 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh" event={"ID":"f150eabb-c9b7-4498-8f06-8f9243d8f058","Type":"ContainerDied","Data":"96da1c398575c819a35899074950b853b021d28972a35d4199d131fca13ee7cb"} Sep 08 04:32:35.277999 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.277922 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh" event={"ID":"f150eabb-c9b7-4498-8f06-8f9243d8f058","Type":"ContainerDied","Data":"6b4431c39b1b023459684751b27fb39fdf33f3c5305ea33115ccf08806d70eec"} Sep 08 04:32:35.285951 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.285932 2574 scope.go:117] "RemoveContainer" containerID="6444adb957bbf186fb5e7d6f36dfaf8dfad0d8e8c252c00f156cc6a80d4f8cb6" Sep 08 04:32:35.286224 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:32:35.286206 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6444adb957bbf186fb5e7d6f36dfaf8dfad0d8e8c252c00f156cc6a80d4f8cb6\": container with ID starting with 6444adb957bbf186fb5e7d6f36dfaf8dfad0d8e8c252c00f156cc6a80d4f8cb6 not found: ID does not exist" containerID="6444adb957bbf186fb5e7d6f36dfaf8dfad0d8e8c252c00f156cc6a80d4f8cb6" Sep 08 04:32:35.286285 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.286232 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6444adb957bbf186fb5e7d6f36dfaf8dfad0d8e8c252c00f156cc6a80d4f8cb6"} err="failed to get container status \"6444adb957bbf186fb5e7d6f36dfaf8dfad0d8e8c252c00f156cc6a80d4f8cb6\": rpc error: code = NotFound desc = could not find container \"6444adb957bbf186fb5e7d6f36dfaf8dfad0d8e8c252c00f156cc6a80d4f8cb6\": container with ID starting with 6444adb957bbf186fb5e7d6f36dfaf8dfad0d8e8c252c00f156cc6a80d4f8cb6 not found: ID does not exist" Sep 08 04:32:35.286285 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.286248 2574 scope.go:117] "RemoveContainer" containerID="96da1c398575c819a35899074950b853b021d28972a35d4199d131fca13ee7cb" Sep 08 04:32:35.294088 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.294072 2574 scope.go:117] "RemoveContainer" containerID="96da1c398575c819a35899074950b853b021d28972a35d4199d131fca13ee7cb" Sep 08 04:32:35.294363 ip-10-0-133-37 kubenswrapper[2574]: E0908 04:32:35.294344 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96da1c398575c819a35899074950b853b021d28972a35d4199d131fca13ee7cb\": container with ID starting with 96da1c398575c819a35899074950b853b021d28972a35d4199d131fca13ee7cb not found: ID does not exist" containerID="96da1c398575c819a35899074950b853b021d28972a35d4199d131fca13ee7cb" Sep 08 04:32:35.294417 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.294370 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96da1c398575c819a35899074950b853b021d28972a35d4199d131fca13ee7cb"} err="failed to get container status \"96da1c398575c819a35899074950b853b021d28972a35d4199d131fca13ee7cb\": rpc error: code = NotFound desc = could not find container \"96da1c398575c819a35899074950b853b021d28972a35d4199d131fca13ee7cb\": container with ID starting with 96da1c398575c819a35899074950b853b021d28972a35d4199d131fca13ee7cb not found: ID does not exist" Sep 08 04:32:35.304339 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.304314 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7"] Sep 08 04:32:35.310198 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.310180 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-720789-55f75d4486-z8dp7"] Sep 08 04:32:35.323467 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.323421 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh"] Sep 08 04:32:35.326740 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.326721 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-720789-5759f8c6c9-27djh"] Sep 08 04:32:35.347094 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.347073 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83d7d5f8-f5da-406a-a976-388737723147-plugins-config-volume\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:32:35.347094 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.347090 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfw2r\" (UniqueName: \"kubernetes.io/projected/83d7d5f8-f5da-406a-a976-388737723147-kube-api-access-sfw2r\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:32:35.347211 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.347101 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f150eabb-c9b7-4498-8f06-8f9243d8f058-plugins-config-volume\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:32:35.347211 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.347110 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlcdr\" (UniqueName: \"kubernetes.io/projected/f150eabb-c9b7-4498-8f06-8f9243d8f058-kube-api-access-mlcdr\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Sep 08 04:32:35.431974 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:35.431933 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:32:36.566560 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:36.566537 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83d7d5f8-f5da-406a-a976-388737723147" path="/var/lib/kubelet/pods/83d7d5f8-f5da-406a-a976-388737723147/volumes" Sep 08 04:32:36.566870 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:36.566793 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeca0803-5381-411b-bda9-ede149a440e7" path="/var/lib/kubelet/pods/aeca0803-5381-411b-bda9-ede149a440e7/volumes" Sep 08 04:32:36.567069 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:36.567057 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f150eabb-c9b7-4498-8f06-8f9243d8f058" path="/var/lib/kubelet/pods/f150eabb-c9b7-4498-8f06-8f9243d8f058/volumes" Sep 08 04:32:41.551973 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:41.551929 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:32:41.624633 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:41.624599 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:32:41.842824 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:32:41.842743 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:33:27.025548 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:33:27.025461 2574 scope.go:117] "RemoveContainer" containerID="021b51a92aeabd30ab1a087130a972de28f13260a66bd66510bf0c5735329eef" Sep 08 04:34:06.006819 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:34:06.006771 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:34:08.401439 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:34:08.401403 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:34:09.924256 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:34:09.924210 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:34:28.043765 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:34:28.043723 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:35:47.104726 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:35:47.104688 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:36:27.081354 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:36:27.081279 2574 scope.go:117] "RemoveContainer" containerID="411fb75597dfac1b3dd6ac86a7d3c91e4f182df074b48c94069e134ba60c9113" Sep 08 04:36:27.089807 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:36:27.089787 2574 scope.go:117] "RemoveContainer" containerID="f09a5564902036f139addb456ba8dae8a30d9d298fb5428a18f60b2a557f5574" Sep 08 04:36:27.099030 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:36:27.099013 2574 scope.go:117] "RemoveContainer" containerID="d02d42dd38d2ccaa8b44b6f07ab7f6631842ca65bf27a8ce20529585bd37d21c" Sep 08 04:36:54.322954 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:36:54.322917 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:37:20.395077 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:37:20.395043 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:37:27.124478 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:37:27.124446 2574 scope.go:117] "RemoveContainer" containerID="28ab9c58386fa66c10837e8f452e10f97c4f64bb3fae2377b737953d33d05d5d" Sep 08 04:37:27.132662 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:37:27.132644 2574 scope.go:117] "RemoveContainer" containerID="825a596b4aee121976f38babc414431d38958bd8cc44a84ebebabd2c38b304e1" Sep 08 04:37:38.871843 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:37:38.871809 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:38:09.308588 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:38:09.308553 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:38:10.571161 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:38:10.571127 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:38:27.154562 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:38:27.154525 2574 scope.go:117] "RemoveContainer" containerID="ab5919b4dfc5a1574065f1058ae118986f08ed7a6a2340bace1a79bc244d6f20" Sep 08 04:38:37.406554 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:38:37.406517 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:38:37.507673 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:38:37.507647 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:39:09.309476 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:09.309432 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:39:10.779847 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:10.779809 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vxz9r_6db0e3e2-e65f-40fb-9150-fd7d087e5de9/server/0.log" Sep 08 04:39:10.914013 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:10.913978 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-nwlcc_f45744aa-dccc-43d2-be9a-dc182ba87805/manager/0.log" Sep 08 04:39:11.169628 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:11.169556 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d5d59f488-6t42t_f718da3d-1d9b-489d-87fe-a241d6f748c1/copy-manifests/0.log" Sep 08 04:39:11.184065 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:11.184040 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d5d59f488-6t42t_f718da3d-1d9b-489d-87fe-a241d6f748c1/manager/0.log" Sep 08 04:39:11.936911 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:11.936877 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29814015-2jplb_4247cb85-2393-4e48-8f8a-5d23d56d9a03/cleanup/0.log" Sep 08 04:39:12.060120 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:12.060089 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29814030-kxj7d_d2eb5f92-6531-4ef5-9d47-4b50d0105454/cleanup/0.log" Sep 08 04:39:14.161603 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:14.161567 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd_2678a7f5-2092-47d8-828f-1dd24f01b9e2/istio-proxy/0.log" Sep 08 04:39:14.285146 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:14.285113 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-p94kg_b0f4a49d-e2d6-4f39-9648-c55322d28666/discovery/0.log" Sep 08 04:39:15.028099 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:15.028071 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55766f96d-krtn9_e1a7f679-2737-437e-a718-53fe941b999a/router/0.log" Sep 08 04:39:15.391480 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:15.391408 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-jpw7l_fad15175-3c67-4e80-8416-a1087cde7b99/main/0.log" Sep 08 04:39:15.505875 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:15.505851 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-fg2fx_a342471e-e7cc-4a0f-9f71-be694b5c88b5/main/0.log" Sep 08 04:39:22.178868 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.178788 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cchdf/must-gather-tx7vf"] Sep 08 04:39:22.179246 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179098 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eb7575b-685f-48e8-8616-b52b6722357f" containerName="payload-processing" Sep 08 04:39:22.179246 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179110 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eb7575b-685f-48e8-8616-b52b6722357f" containerName="payload-processing" Sep 08 04:39:22.179246 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179119 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc67f27e-765a-425a-9b20-d7d05f43301c" containerName="istio-proxy" Sep 08 04:39:22.179246 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179125 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc67f27e-765a-425a-9b20-d7d05f43301c" containerName="istio-proxy" Sep 08 04:39:22.179246 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179134 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc67f27e-765a-425a-9b20-d7d05f43301c" containerName="istio-proxy" Sep 08 04:39:22.179246 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179141 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc67f27e-765a-425a-9b20-d7d05f43301c" containerName="istio-proxy" Sep 08 04:39:22.179246 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179155 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55" containerName="istio-proxy" Sep 08 04:39:22.179246 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179161 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55" containerName="istio-proxy" Sep 08 04:39:22.179246 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179167 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="247e8db0-c1bd-4fa6-ac02-105339ab529e" containerName="revoke-keys" Sep 08 04:39:22.179246 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179172 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="247e8db0-c1bd-4fa6-ac02-105339ab529e" containerName="revoke-keys" Sep 08 04:39:22.179246 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179180 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3427054f-6ed2-482b-817a-8f6859f7c383" containerName="istio-proxy" Sep 08 04:39:22.179246 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179185 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3427054f-6ed2-482b-817a-8f6859f7c383" containerName="istio-proxy" Sep 08 04:39:22.179246 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179192 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3427054f-6ed2-482b-817a-8f6859f7c383" containerName="istio-proxy" Sep 08 04:39:22.179246 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179197 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3427054f-6ed2-482b-817a-8f6859f7c383" containerName="istio-proxy" Sep 08 04:39:22.179246 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179204 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f150eabb-c9b7-4498-8f06-8f9243d8f058" containerName="payload-processing" Sep 08 04:39:22.179246 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179209 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f150eabb-c9b7-4498-8f06-8f9243d8f058" containerName="payload-processing" Sep 08 04:39:22.179246 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179219 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83d7d5f8-f5da-406a-a976-388737723147" containerName="payload-pre-processing" Sep 08 04:39:22.179246 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179224 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="83d7d5f8-f5da-406a-a976-388737723147" containerName="payload-pre-processing" Sep 08 04:39:22.179999 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179279 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eb7575b-685f-48e8-8616-b52b6722357f" containerName="payload-processing" Sep 08 04:39:22.179999 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179287 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="83d7d5f8-f5da-406a-a976-388737723147" containerName="payload-pre-processing" Sep 08 04:39:22.179999 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179293 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55" containerName="istio-proxy" Sep 08 04:39:22.179999 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179298 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55" containerName="istio-proxy" Sep 08 04:39:22.179999 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179304 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc67f27e-765a-425a-9b20-d7d05f43301c" containerName="istio-proxy" Sep 08 04:39:22.179999 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179310 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc67f27e-765a-425a-9b20-d7d05f43301c" containerName="istio-proxy" Sep 08 04:39:22.179999 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179316 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3427054f-6ed2-482b-817a-8f6859f7c383" containerName="istio-proxy" Sep 08 04:39:22.179999 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179320 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f150eabb-c9b7-4498-8f06-8f9243d8f058" containerName="payload-processing" Sep 08 04:39:22.179999 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179326 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3427054f-6ed2-482b-817a-8f6859f7c383" containerName="istio-proxy" Sep 08 04:39:22.179999 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179330 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="247e8db0-c1bd-4fa6-ac02-105339ab529e" containerName="revoke-keys" Sep 08 04:39:22.179999 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179377 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55" containerName="istio-proxy" Sep 08 04:39:22.179999 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.179383 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dae9cb8-8fe8-482b-a2e5-a9120ea4ef55" containerName="istio-proxy" Sep 08 04:39:22.182471 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.182453 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cchdf/must-gather-tx7vf" Sep 08 04:39:22.185083 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.185059 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cchdf\"/\"default-dockercfg-lk299\"" Sep 08 04:39:22.185083 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.185061 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cchdf\"/\"kube-root-ca.crt\"" Sep 08 04:39:22.185398 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.185380 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cchdf\"/\"openshift-service-ca.crt\"" Sep 08 04:39:22.196026 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.196002 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cchdf/must-gather-tx7vf"] Sep 08 04:39:22.212106 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.212085 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:39:22.243329 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.243308 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5rrn\" (UniqueName: \"kubernetes.io/projected/fb4037b4-dab0-4fa5-8184-84d461e71141-kube-api-access-f5rrn\") pod \"must-gather-tx7vf\" (UID: \"fb4037b4-dab0-4fa5-8184-84d461e71141\") " pod="openshift-must-gather-cchdf/must-gather-tx7vf" Sep 08 04:39:22.243418 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.243350 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fb4037b4-dab0-4fa5-8184-84d461e71141-must-gather-output\") pod \"must-gather-tx7vf\" (UID: \"fb4037b4-dab0-4fa5-8184-84d461e71141\") " pod="openshift-must-gather-cchdf/must-gather-tx7vf" Sep 08 04:39:22.298765 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.298739 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" limit=1024 Sep 08 04:39:22.344282 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.344261 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fb4037b4-dab0-4fa5-8184-84d461e71141-must-gather-output\") pod \"must-gather-tx7vf\" (UID: \"fb4037b4-dab0-4fa5-8184-84d461e71141\") " pod="openshift-must-gather-cchdf/must-gather-tx7vf" Sep 08 04:39:22.344358 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.344316 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5rrn\" (UniqueName: \"kubernetes.io/projected/fb4037b4-dab0-4fa5-8184-84d461e71141-kube-api-access-f5rrn\") pod \"must-gather-tx7vf\" (UID: \"fb4037b4-dab0-4fa5-8184-84d461e71141\") " pod="openshift-must-gather-cchdf/must-gather-tx7vf" Sep 08 04:39:22.344587 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.344569 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fb4037b4-dab0-4fa5-8184-84d461e71141-must-gather-output\") pod \"must-gather-tx7vf\" (UID: \"fb4037b4-dab0-4fa5-8184-84d461e71141\") " pod="openshift-must-gather-cchdf/must-gather-tx7vf" Sep 08 04:39:22.352311 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.352292 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5rrn\" (UniqueName: \"kubernetes.io/projected/fb4037b4-dab0-4fa5-8184-84d461e71141-kube-api-access-f5rrn\") pod \"must-gather-tx7vf\" (UID: \"fb4037b4-dab0-4fa5-8184-84d461e71141\") " pod="openshift-must-gather-cchdf/must-gather-tx7vf" Sep 08 04:39:22.492102 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.492050 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cchdf/must-gather-tx7vf" Sep 08 04:39:22.762918 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.762895 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cchdf/must-gather-tx7vf"] Sep 08 04:39:22.764868 ip-10-0-133-37 kubenswrapper[2574]: W0908 04:39:22.764837 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb4037b4_dab0_4fa5_8184_84d461e71141.slice/crio-c34de5e7e980cd1c40bbeb5814bdf13bb44e4cbe887788768c0c6564933db24a WatchSource:0}: Error finding container c34de5e7e980cd1c40bbeb5814bdf13bb44e4cbe887788768c0c6564933db24a: Status 404 returned error can't find the container with id c34de5e7e980cd1c40bbeb5814bdf13bb44e4cbe887788768c0c6564933db24a Sep 08 04:39:22.766634 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:22.766613 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 04:39:23.541595 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:23.541545 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cchdf/must-gather-tx7vf" event={"ID":"fb4037b4-dab0-4fa5-8184-84d461e71141","Type":"ContainerStarted","Data":"687339fb187a5e2a3ef45c7b38598f9f7933390de3d46e5dd7e058a65eb7d9e3"} Sep 08 04:39:23.541595 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:23.541599 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cchdf/must-gather-tx7vf" event={"ID":"fb4037b4-dab0-4fa5-8184-84d461e71141","Type":"ContainerStarted","Data":"c34de5e7e980cd1c40bbeb5814bdf13bb44e4cbe887788768c0c6564933db24a"} Sep 08 04:39:24.555800 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:24.555761 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cchdf/must-gather-tx7vf" event={"ID":"fb4037b4-dab0-4fa5-8184-84d461e71141","Type":"ContainerStarted","Data":"2b69bab17cc1a987fd4d4311ca6af853f9fc7cfc5b20532a0eb8f58c13c73f4c"} Sep 08 04:39:24.571101 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:24.571055 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cchdf/must-gather-tx7vf" podStartSLOduration=1.933853195 podStartE2EDuration="2.571040933s" totalImagesPullingTime="637.187738ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 04:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:39:24.569710048 +0000 UTC m=+2338.492170914" watchObservedRunningTime="2026-09-08 04:39:24.571040933 +0000 UTC m=+2338.493501670" Sep 08 04:39:24.974558 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:24.974474 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x4dkl_c075d89b-4d37-4e76-a9e5-d61c6e9cf2eb/global-pull-secret-syncer/0.log" Sep 08 04:39:25.023229 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:25.023202 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-84pxf_fab1c08f-5e53-42c6-a080-e510ae257469/konnectivity-agent/0.log" Sep 08 04:39:25.135271 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:25.135241 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-37.ec2.internal_480faa79f347b524a5131c7d80428e3a/haproxy/0.log" Sep 08 04:39:31.409214 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:31.409167 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-pqx2g_b35d6e27-f1ad-4481-a512-36851544fffa/cluster-monitoring-operator/0.log" Sep 08 04:39:31.651527 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:31.649040 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7mx8j_4b15a3b0-00f6-419e-91b5-f0cc4c166d17/node-exporter/0.log" Sep 08 04:39:31.689594 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:31.689571 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7mx8j_4b15a3b0-00f6-419e-91b5-f0cc4c166d17/kube-rbac-proxy/0.log" Sep 08 04:39:31.713387 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:31.713361 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7mx8j_4b15a3b0-00f6-419e-91b5-f0cc4c166d17/init-textfile/0.log" Sep 08 04:39:33.836110 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:33.836080 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-6czj4_d0fb6dc7-9787-40e0-a9d4-fe3a16f34750/networking-console-plugin/0.log" Sep 08 04:39:34.838356 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:34.838319 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cchdf/perf-node-gather-daemonset-7kfcb"] Sep 08 04:39:34.852065 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:34.850808 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cchdf/perf-node-gather-daemonset-7kfcb"] Sep 08 04:39:34.852065 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:34.850957 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-7kfcb" Sep 08 04:39:34.965090 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:34.965047 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/edfd3bcb-ca2e-4163-befe-d0e8326fa6ee-podres\") pod \"perf-node-gather-daemonset-7kfcb\" (UID: \"edfd3bcb-ca2e-4163-befe-d0e8326fa6ee\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-7kfcb" Sep 08 04:39:34.965285 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:34.965127 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/edfd3bcb-ca2e-4163-befe-d0e8326fa6ee-proc\") pod \"perf-node-gather-daemonset-7kfcb\" (UID: \"edfd3bcb-ca2e-4163-befe-d0e8326fa6ee\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-7kfcb" Sep 08 04:39:34.965285 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:34.965176 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/edfd3bcb-ca2e-4163-befe-d0e8326fa6ee-sys\") pod \"perf-node-gather-daemonset-7kfcb\" (UID: \"edfd3bcb-ca2e-4163-befe-d0e8326fa6ee\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-7kfcb" Sep 08 04:39:34.965285 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:34.965226 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qjds\" (UniqueName: \"kubernetes.io/projected/edfd3bcb-ca2e-4163-befe-d0e8326fa6ee-kube-api-access-2qjds\") pod \"perf-node-gather-daemonset-7kfcb\" (UID: \"edfd3bcb-ca2e-4163-befe-d0e8326fa6ee\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-7kfcb" Sep 08 04:39:34.965427 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:34.965284 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/edfd3bcb-ca2e-4163-befe-d0e8326fa6ee-lib-modules\") pod \"perf-node-gather-daemonset-7kfcb\" (UID: \"edfd3bcb-ca2e-4163-befe-d0e8326fa6ee\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-7kfcb" Sep 08 04:39:35.066289 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:35.066253 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/edfd3bcb-ca2e-4163-befe-d0e8326fa6ee-podres\") pod \"perf-node-gather-daemonset-7kfcb\" (UID: \"edfd3bcb-ca2e-4163-befe-d0e8326fa6ee\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-7kfcb" Sep 08 04:39:35.066493 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:35.066311 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/edfd3bcb-ca2e-4163-befe-d0e8326fa6ee-proc\") pod \"perf-node-gather-daemonset-7kfcb\" (UID: \"edfd3bcb-ca2e-4163-befe-d0e8326fa6ee\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-7kfcb" Sep 08 04:39:35.066493 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:35.066348 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/edfd3bcb-ca2e-4163-befe-d0e8326fa6ee-sys\") pod \"perf-node-gather-daemonset-7kfcb\" (UID: \"edfd3bcb-ca2e-4163-befe-d0e8326fa6ee\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-7kfcb" Sep 08 04:39:35.066493 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:35.066427 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/edfd3bcb-ca2e-4163-befe-d0e8326fa6ee-sys\") pod \"perf-node-gather-daemonset-7kfcb\" (UID: \"edfd3bcb-ca2e-4163-befe-d0e8326fa6ee\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-7kfcb" Sep 08 04:39:35.066493 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:35.066427 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/edfd3bcb-ca2e-4163-befe-d0e8326fa6ee-podres\") pod \"perf-node-gather-daemonset-7kfcb\" (UID: \"edfd3bcb-ca2e-4163-befe-d0e8326fa6ee\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-7kfcb" Sep 08 04:39:35.066493 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:35.066439 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/edfd3bcb-ca2e-4163-befe-d0e8326fa6ee-proc\") pod \"perf-node-gather-daemonset-7kfcb\" (UID: \"edfd3bcb-ca2e-4163-befe-d0e8326fa6ee\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-7kfcb" Sep 08 04:39:35.066493 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:35.066458 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qjds\" (UniqueName: \"kubernetes.io/projected/edfd3bcb-ca2e-4163-befe-d0e8326fa6ee-kube-api-access-2qjds\") pod \"perf-node-gather-daemonset-7kfcb\" (UID: \"edfd3bcb-ca2e-4163-befe-d0e8326fa6ee\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-7kfcb" Sep 08 04:39:35.066851 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:35.066550 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/edfd3bcb-ca2e-4163-befe-d0e8326fa6ee-lib-modules\") pod \"perf-node-gather-daemonset-7kfcb\" (UID: \"edfd3bcb-ca2e-4163-befe-d0e8326fa6ee\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-7kfcb" Sep 08 04:39:35.066851 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:35.066671 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/edfd3bcb-ca2e-4163-befe-d0e8326fa6ee-lib-modules\") pod \"perf-node-gather-daemonset-7kfcb\" (UID: \"edfd3bcb-ca2e-4163-befe-d0e8326fa6ee\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-7kfcb" Sep 08 04:39:35.074389 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:35.074363 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qjds\" (UniqueName: \"kubernetes.io/projected/edfd3bcb-ca2e-4163-befe-d0e8326fa6ee-kube-api-access-2qjds\") pod \"perf-node-gather-daemonset-7kfcb\" (UID: \"edfd3bcb-ca2e-4163-befe-d0e8326fa6ee\") " pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-7kfcb" Sep 08 04:39:35.077445 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:35.077402 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-69cd8577c5-qwb8s_afa004f7-56b9-4b11-a01d-f14fb3491d06/console/0.log" Sep 08 04:39:35.176349 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:35.176246 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-7kfcb" Sep 08 04:39:35.475546 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:35.473033 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cchdf/perf-node-gather-daemonset-7kfcb"] Sep 08 04:39:35.657219 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:35.657169 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-7kfcb" event={"ID":"edfd3bcb-ca2e-4163-befe-d0e8326fa6ee","Type":"ContainerStarted","Data":"ed3e982cbff21aa8d707a47672489bc2f49ae132605df1e15f12917deb947e69"} Sep 08 04:39:35.657418 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:35.657270 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-7kfcb" event={"ID":"edfd3bcb-ca2e-4163-befe-d0e8326fa6ee","Type":"ContainerStarted","Data":"2a2964467bf3c87943fb8b62912408688684ce70232ae784d416cdf6a82e7773"} Sep 08 04:39:35.657518 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:35.657483 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-7kfcb" Sep 08 04:39:35.673801 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:35.673746 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-7kfcb" podStartSLOduration=1.6737251610000001 podStartE2EDuration="1.673725161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 04:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 04:39:35.671294441 +0000 UTC m=+2349.593755198" watchObservedRunningTime="2026-09-08 04:39:35.673725161 +0000 UTC m=+2349.596185905" Sep 08 04:39:36.921373 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:36.921347 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b8vzf_57a0643d-28d1-4d11-bb40-1de33627ca6d/dns/0.log" Sep 08 04:39:36.942348 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:36.942323 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b8vzf_57a0643d-28d1-4d11-bb40-1de33627ca6d/kube-rbac-proxy/0.log" Sep 08 04:39:37.055965 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:37.055942 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-drx9g_f1262da6-55b5-49dd-8b8b-187c86dc46f6/dns-node-resolver/0.log" Sep 08 04:39:37.628430 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:37.628398 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-64d47bbb76-26f2f_2cf49095-f078-4f11-a993-697cebff6172/registry/0.log" Sep 08 04:39:37.676237 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:37.676210 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qcbd8_6698d58b-300e-4f04-a773-1756fcd6f558/node-ca/0.log" Sep 08 04:39:38.602578 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:38.602523 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd_2678a7f5-2092-47d8-828f-1dd24f01b9e2/istio-proxy/0.log" Sep 08 04:39:38.990788 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:38.990717 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-p94kg_b0f4a49d-e2d6-4f39-9648-c55322d28666/discovery/0.log" Sep 08 04:39:39.161832 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:39.161801 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55766f96d-krtn9_e1a7f679-2737-437e-a718-53fe941b999a/router/0.log" Sep 08 04:39:40.070746 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:40.070717 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pv6qv_3e2eec1f-b2a7-43d5-a9e7-f15eafb23937/serve-healthcheck-canary/0.log" Sep 08 04:39:40.632789 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:40.632749 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kjg2p_5b5cda57-838f-4ba0-87be-f09808d06d65/kube-rbac-proxy/0.log" Sep 08 04:39:40.655796 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:40.655771 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kjg2p_5b5cda57-838f-4ba0-87be-f09808d06d65/exporter/0.log" Sep 08 04:39:40.679251 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:40.679212 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kjg2p_5b5cda57-838f-4ba0-87be-f09808d06d65/extractor/0.log" Sep 08 04:39:41.673642 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:41.673615 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cchdf/perf-node-gather-daemonset-7kfcb" Sep 08 04:39:43.054535 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:43.054477 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vxz9r_6db0e3e2-e65f-40fb-9150-fd7d087e5de9/server/0.log" Sep 08 04:39:43.157484 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:43.157447 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-nwlcc_f45744aa-dccc-43d2-be9a-dc182ba87805/manager/0.log" Sep 08 04:39:43.326838 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:43.326749 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d5d59f488-6t42t_f718da3d-1d9b-489d-87fe-a241d6f748c1/manager/0.log" Sep 08 04:39:43.345373 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:43.345341 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d5d59f488-6t42t_f718da3d-1d9b-489d-87fe-a241d6f748c1/copy-manifests/0.log" Sep 08 04:39:44.858396 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:44.858369 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-58d7df9668-hgqgf_c5b842e7-f15f-4453-bb7e-8191c1951874/manager/0.log" Sep 08 04:39:44.889415 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:44.889379 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-9w4v5_e10c809c-a846-48c6-85a6-ca6554bcb5fe/openshift-lws-operator/0.log" Sep 08 04:39:45.064867 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:45.064840 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd_2678a7f5-2092-47d8-828f-1dd24f01b9e2/istio-proxy/0.log" Sep 08 04:39:45.449011 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:45.448933 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-p94kg_b0f4a49d-e2d6-4f39-9648-c55322d28666/discovery/0.log" Sep 08 04:39:45.566928 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:45.566898 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55766f96d-krtn9_e1a7f679-2737-437e-a718-53fe941b999a/router/0.log" Sep 08 04:39:47.035242 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:47.035100 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-6fpjd_2678a7f5-2092-47d8-828f-1dd24f01b9e2/istio-proxy/0.log" Sep 08 04:39:47.422848 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:47.422764 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-p94kg_b0f4a49d-e2d6-4f39-9648-c55322d28666/discovery/0.log" Sep 08 04:39:47.556390 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:47.556361 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55766f96d-krtn9_e1a7f679-2737-437e-a718-53fe941b999a/router/0.log" Sep 08 04:39:50.407539 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:50.407484 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-xmcbr_56994db0-6aa0-44a7-a540-777271c022ee/kube-storage-version-migrator-operator/1.log" Sep 08 04:39:50.408450 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:50.408432 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-xmcbr_56994db0-6aa0-44a7-a540-777271c022ee/kube-storage-version-migrator-operator/0.log" Sep 08 04:39:51.700038 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:51.700013 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nrhb8_8ac44940-1e7f-48c1-a9db-8ac81a574f45/kube-multus-additional-cni-plugins/0.log" Sep 08 04:39:51.727003 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:51.726981 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nrhb8_8ac44940-1e7f-48c1-a9db-8ac81a574f45/egress-router-binary-copy/0.log" Sep 08 04:39:51.752438 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:51.752420 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nrhb8_8ac44940-1e7f-48c1-a9db-8ac81a574f45/cni-plugins/0.log" Sep 08 04:39:51.779156 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:51.779127 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nrhb8_8ac44940-1e7f-48c1-a9db-8ac81a574f45/bond-cni-plugin/0.log" Sep 08 04:39:51.802768 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:51.802750 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nrhb8_8ac44940-1e7f-48c1-a9db-8ac81a574f45/routeoverride-cni/0.log" Sep 08 04:39:51.825393 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:51.825371 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nrhb8_8ac44940-1e7f-48c1-a9db-8ac81a574f45/whereabouts-cni-bincopy/0.log" Sep 08 04:39:51.846303 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:51.846283 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nrhb8_8ac44940-1e7f-48c1-a9db-8ac81a574f45/whereabouts-cni/0.log" Sep 08 04:39:51.897208 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:51.897191 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-d95fg_ce867bba-0058-4825-bad0-6962d3ea4890/kube-multus/0.log" Sep 08 04:39:52.064339 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:52.064310 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rzwvc_ff12a182-ed89-4513-938b-69971199ad25/network-metrics-daemon/0.log" Sep 08 04:39:52.087875 ip-10-0-133-37 kubenswrapper[2574]: I0908 04:39:52.087845 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rzwvc_ff12a182-ed89-4513-938b-69971199ad25/kube-rbac-proxy/0.log"