Sep 10 23:34:40.348783 ip-10-0-139-142 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 10 23:34:40.348794 ip-10-0-139-142 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 10 23:34:40.348804 ip-10-0-139-142 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 10 23:34:40.349366 ip-10-0-139-142 systemd[1]: Failed to start Kubernetes Kubelet. Sep 10 23:34:50.516972 ip-10-0-139-142 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 10 23:34:50.516993 ip-10-0-139-142 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 183235b9d77b470ea705fc5d0fd6f51a -- Sep 10 23:36:23.262687 ip-10-0-139-142 systemd[1]: Starting Kubernetes Kubelet... Sep 10 23:36:23.679271 ip-10-0-139-142 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 23:36:23.679271 ip-10-0-139-142 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 23:36:23.679271 ip-10-0-139-142 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 23:36:23.679271 ip-10-0-139-142 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 23:36:23.679271 ip-10-0-139-142 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 23:36:23.681231 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.681141 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 23:36:23.684241 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684219 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 23:36:23.684241 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684238 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 23:36:23.684241 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684243 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 23:36:23.684241 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684247 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 23:36:23.684535 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684253 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 23:36:23.684535 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684257 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 23:36:23.684535 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684261 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 23:36:23.684535 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684265 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 23:36:23.684535 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684269 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 23:36:23.684535 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684272 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 23:36:23.684535 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684276 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 23:36:23.684535 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684280 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 23:36:23.684535 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684284 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 23:36:23.684535 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684287 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 23:36:23.684535 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684291 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 23:36:23.684535 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684300 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 23:36:23.684535 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684305 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 23:36:23.684535 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684308 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 23:36:23.684535 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684312 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 23:36:23.684535 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684316 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 23:36:23.684535 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684321 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 23:36:23.684535 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684325 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 23:36:23.684535 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684329 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 23:36:23.684535 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684332 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 23:36:23.685351 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684336 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 23:36:23.685351 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684340 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 23:36:23.685351 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684345 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 23:36:23.685351 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684349 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 23:36:23.685351 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684353 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 23:36:23.685351 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684357 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 23:36:23.685351 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684361 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 23:36:23.685351 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684365 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 23:36:23.685351 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684384 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 23:36:23.685351 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684389 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 23:36:23.685351 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684393 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 23:36:23.685351 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684397 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 23:36:23.685351 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684401 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 23:36:23.685351 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684406 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 23:36:23.685351 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684410 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 23:36:23.685351 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684414 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 23:36:23.685351 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684418 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 23:36:23.685351 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684424 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 23:36:23.685351 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684429 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 23:36:23.685351 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684433 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 23:36:23.686256 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684438 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 23:36:23.686256 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684443 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 23:36:23.686256 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684447 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 23:36:23.686256 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684451 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 23:36:23.686256 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684456 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 23:36:23.686256 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684461 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 23:36:23.686256 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684465 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 23:36:23.686256 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684474 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 23:36:23.686256 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684480 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 23:36:23.686256 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684485 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 23:36:23.686256 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684490 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 23:36:23.686256 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684495 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 23:36:23.686256 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684499 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 23:36:23.686256 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684503 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 23:36:23.686256 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684508 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 23:36:23.686256 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684513 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 23:36:23.686256 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684517 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 10 23:36:23.686256 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684521 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 23:36:23.686256 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684526 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 23:36:23.686256 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684529 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 23:36:23.687104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684533 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 23:36:23.687104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684537 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 23:36:23.687104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684541 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 23:36:23.687104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684546 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 23:36:23.687104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684552 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 23:36:23.687104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684557 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 23:36:23.687104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684562 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 23:36:23.687104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684569 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 23:36:23.687104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684574 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 23:36:23.687104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684579 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 23:36:23.687104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684583 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 23:36:23.687104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684587 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 23:36:23.687104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684591 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 23:36:23.687104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684595 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 23:36:23.687104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684600 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 23:36:23.687104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684604 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 23:36:23.687104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684608 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 23:36:23.687104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684619 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 23:36:23.687104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684623 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 23:36:23.687922 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684628 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 23:36:23.687922 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684636 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 23:36:23.687922 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684641 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 23:36:23.687922 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684646 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 23:36:23.687922 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684650 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 23:36:23.687922 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684655 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 23:36:23.687922 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684661 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 23:36:23.687922 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684665 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 23:36:23.687922 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684670 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 23:36:23.687922 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684674 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 23:36:23.687922 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684679 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 23:36:23.687922 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684683 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 23:36:23.687922 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684689 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 23:36:23.687922 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684693 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 23:36:23.687922 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684697 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 23:36:23.687922 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.684702 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 23:36:23.687922 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.685894 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 23:36:23.687922 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.685904 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 23:36:23.687922 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.685909 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 23:36:23.688916 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.685914 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 23:36:23.688916 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.685919 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 23:36:23.688916 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.685924 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 23:36:23.688916 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.685928 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 23:36:23.688916 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.685932 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 23:36:23.688916 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.685936 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 23:36:23.688916 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.685941 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 23:36:23.688916 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.685945 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 23:36:23.688916 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.685950 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 23:36:23.688916 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.685955 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 23:36:23.688916 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.685960 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 23:36:23.688916 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.685964 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 23:36:23.688916 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.685970 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 23:36:23.688916 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.685974 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 23:36:23.688916 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.685978 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 23:36:23.688916 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.685983 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 23:36:23.688916 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.685987 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 23:36:23.688916 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.685991 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 23:36:23.688916 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.685995 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 23:36:23.689558 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686002 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 23:36:23.689558 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686009 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 23:36:23.689558 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686014 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 23:36:23.689558 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686019 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 23:36:23.689558 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686024 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 23:36:23.689558 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686029 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 23:36:23.689558 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686034 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 23:36:23.689558 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686038 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 23:36:23.689558 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686043 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 23:36:23.689558 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686048 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 23:36:23.689558 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686052 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 23:36:23.689558 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686056 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 23:36:23.689558 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686062 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 23:36:23.689558 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686066 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 23:36:23.689558 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686070 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 23:36:23.689558 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686074 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 23:36:23.689558 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686078 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 23:36:23.689558 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686084 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 23:36:23.689558 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686088 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 23:36:23.690104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686093 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 23:36:23.690104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686097 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 23:36:23.690104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686102 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 23:36:23.690104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686106 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 23:36:23.690104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686110 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 23:36:23.690104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686114 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 23:36:23.690104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686118 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 23:36:23.690104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686123 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 23:36:23.690104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686128 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 10 23:36:23.690104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686132 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 23:36:23.690104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686136 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 23:36:23.690104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686141 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 23:36:23.690104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686145 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 23:36:23.690104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686149 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 23:36:23.690104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686153 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 23:36:23.690104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686157 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 23:36:23.690104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686162 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 23:36:23.690104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686166 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 23:36:23.690104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686170 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 23:36:23.690104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686175 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 23:36:23.690661 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686182 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 23:36:23.690661 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686188 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 23:36:23.690661 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686193 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 23:36:23.690661 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686197 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 23:36:23.690661 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686202 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 23:36:23.690661 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686207 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 23:36:23.690661 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686212 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 23:36:23.690661 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686217 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 23:36:23.690661 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686222 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 23:36:23.690661 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686226 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 23:36:23.690661 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686231 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 23:36:23.690661 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686235 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 23:36:23.690661 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686249 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 23:36:23.690661 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686254 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 23:36:23.690661 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686259 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 23:36:23.690661 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686263 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 23:36:23.690661 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686267 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 23:36:23.690661 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686271 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 23:36:23.690661 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686275 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 23:36:23.691120 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686279 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 23:36:23.691120 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686284 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 23:36:23.691120 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686288 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 23:36:23.691120 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686293 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 23:36:23.691120 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686297 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 23:36:23.691120 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686301 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 23:36:23.691120 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686306 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 23:36:23.691120 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686310 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 23:36:23.691120 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686314 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 23:36:23.691120 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686318 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 23:36:23.691120 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686323 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 23:36:23.691120 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686328 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 23:36:23.691120 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686332 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 23:36:23.691120 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686337 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 23:36:23.691120 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686341 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 23:36:23.691120 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686345 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 23:36:23.691120 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686350 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 23:36:23.691120 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686354 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 23:36:23.691120 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686359 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 23:36:23.691120 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686986 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 23:36:23.691716 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686994 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 23:36:23.691716 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.686999 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 23:36:23.691716 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687004 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 23:36:23.691716 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687008 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 23:36:23.691716 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687016 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 23:36:23.691716 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687020 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 23:36:23.691716 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687025 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 23:36:23.691716 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687029 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 23:36:23.691716 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687033 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 23:36:23.691716 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687038 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 23:36:23.691716 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687042 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 23:36:23.691716 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687047 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 23:36:23.691716 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687051 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 23:36:23.691716 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687056 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 23:36:23.691716 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687060 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 23:36:23.691716 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687064 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 23:36:23.691716 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687069 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 23:36:23.691716 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687073 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 23:36:23.691716 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687077 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 23:36:23.692188 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687081 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 23:36:23.692188 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687085 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 23:36:23.692188 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687089 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 23:36:23.692188 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687094 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 23:36:23.692188 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687100 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 23:36:23.692188 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687104 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 23:36:23.692188 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687109 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 23:36:23.692188 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687113 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 23:36:23.692188 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687117 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 23:36:23.692188 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687121 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 23:36:23.692188 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687124 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 23:36:23.692188 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687130 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 23:36:23.692188 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687134 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 23:36:23.692188 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687138 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 23:36:23.692188 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687144 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 23:36:23.692188 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687150 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 23:36:23.692188 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687155 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 23:36:23.692188 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687159 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 23:36:23.692188 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687164 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 23:36:23.692188 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687168 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 23:36:23.692687 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687172 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 23:36:23.692687 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687176 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 23:36:23.692687 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687180 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 23:36:23.692687 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687185 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 23:36:23.692687 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687189 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 23:36:23.692687 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687193 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 23:36:23.692687 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687200 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 23:36:23.692687 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687206 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 23:36:23.692687 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687211 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 23:36:23.692687 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687216 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 23:36:23.692687 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687222 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 23:36:23.692687 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687226 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 23:36:23.692687 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687230 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 23:36:23.692687 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687235 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 23:36:23.692687 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687239 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 23:36:23.692687 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687243 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 23:36:23.692687 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687249 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 23:36:23.692687 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687253 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 23:36:23.692687 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687257 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 23:36:23.693143 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687261 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 23:36:23.693143 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687265 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 23:36:23.693143 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687269 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 23:36:23.693143 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687274 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 23:36:23.693143 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687278 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 23:36:23.693143 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687282 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 23:36:23.693143 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687286 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 23:36:23.693143 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687291 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 23:36:23.693143 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687295 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 23:36:23.693143 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687300 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 23:36:23.693143 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687304 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 23:36:23.693143 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687308 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 23:36:23.693143 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687312 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 23:36:23.693143 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687316 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 23:36:23.693143 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687321 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 23:36:23.693143 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687325 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 23:36:23.693143 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687329 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 23:36:23.693143 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687333 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 23:36:23.693143 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687338 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 23:36:23.693143 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687342 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 23:36:23.693646 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687352 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 23:36:23.693646 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687358 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 23:36:23.693646 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687364 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 23:36:23.693646 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687385 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 23:36:23.693646 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687390 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 23:36:23.693646 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687394 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 23:36:23.693646 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687398 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 23:36:23.693646 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687403 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 23:36:23.693646 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687407 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 23:36:23.693646 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687412 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 23:36:23.693646 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687417 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 23:36:23.693646 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687421 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 23:36:23.693646 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687425 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 23:36:23.693646 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687429 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 23:36:23.693646 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687433 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 23:36:23.693646 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687438 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 10 23:36:23.693646 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687442 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 23:36:23.693646 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687446 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 23:36:23.693646 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687450 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 23:36:23.694118 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.687454 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 23:36:23.694118 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687565 2578 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 23:36:23.694118 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687617 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 23:36:23.694118 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687627 2578 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 23:36:23.694118 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687634 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 23:36:23.694118 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687641 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 23:36:23.694118 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687647 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 23:36:23.694118 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687654 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 23:36:23.694118 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687664 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 23:36:23.694118 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687670 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 23:36:23.694118 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687728 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 23:36:23.694118 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687736 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 23:36:23.694118 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687742 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 23:36:23.694118 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687750 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 23:36:23.694118 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687755 2578 flags.go:64] FLAG: --cgroup-root="" Sep 10 23:36:23.694118 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687761 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 23:36:23.694118 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687766 2578 flags.go:64] FLAG: --client-ca-file="" Sep 10 23:36:23.694118 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687771 2578 flags.go:64] FLAG: --cloud-provider="external" Sep 10 23:36:23.694118 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687776 2578 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 23:36:23.694118 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687783 2578 flags.go:64] FLAG: --cluster-domain="" Sep 10 23:36:23.694118 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687787 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 23:36:23.694118 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687793 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 23:36:23.694118 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687799 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 23:36:23.694118 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687805 2578 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 23:36:23.694709 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687812 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 23:36:23.694709 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687817 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 23:36:23.694709 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687822 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 23:36:23.694709 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687828 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 23:36:23.694709 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687833 2578 flags.go:64] FLAG: --contention-profiling="false" Sep 10 23:36:23.694709 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687837 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 23:36:23.694709 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687842 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 23:36:23.694709 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687847 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 23:36:23.694709 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687852 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 23:36:23.694709 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687859 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 23:36:23.694709 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687864 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 23:36:23.694709 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687869 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 23:36:23.694709 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687873 2578 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 23:36:23.694709 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687878 2578 flags.go:64] FLAG: --enable-server="true" Sep 10 23:36:23.694709 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687883 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 23:36:23.694709 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687890 2578 flags.go:64] FLAG: --event-burst="100" Sep 10 23:36:23.694709 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687894 2578 flags.go:64] FLAG: --event-qps="50" Sep 10 23:36:23.694709 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687899 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 23:36:23.694709 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687904 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 23:36:23.694709 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687909 2578 flags.go:64] FLAG: --eviction-hard="" Sep 10 23:36:23.694709 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687925 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 23:36:23.694709 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687932 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 23:36:23.694709 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687938 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 23:36:23.694709 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687942 2578 flags.go:64] FLAG: --eviction-soft="" Sep 10 23:36:23.694709 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687948 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 23:36:23.695330 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687953 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 23:36:23.695330 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687957 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 23:36:23.695330 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687962 2578 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 23:36:23.695330 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687967 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 23:36:23.695330 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687972 2578 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 23:36:23.695330 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687977 2578 flags.go:64] FLAG: --feature-gates="" Sep 10 23:36:23.695330 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687984 2578 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 23:36:23.695330 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687988 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 23:36:23.695330 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687994 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 23:36:23.695330 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.687999 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 23:36:23.695330 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688004 2578 flags.go:64] FLAG: --healthz-port="10248" Sep 10 23:36:23.695330 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688009 2578 flags.go:64] FLAG: --help="false" Sep 10 23:36:23.695330 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688014 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-139-142.ec2.internal" Sep 10 23:36:23.695330 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688019 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 23:36:23.695330 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688023 2578 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 23:36:23.695330 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688028 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 23:36:23.695330 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688033 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 23:36:23.695330 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688039 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 23:36:23.695330 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688043 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 23:36:23.695330 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688048 2578 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 23:36:23.695330 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688053 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 23:36:23.695330 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688057 2578 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 23:36:23.695330 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688062 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 23:36:23.695953 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688067 2578 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 23:36:23.695953 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688071 2578 flags.go:64] FLAG: --kube-reserved="" Sep 10 23:36:23.695953 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688076 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 23:36:23.695953 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688080 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 23:36:23.695953 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688086 2578 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 23:36:23.695953 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688091 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 23:36:23.695953 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688095 2578 flags.go:64] FLAG: --lock-file="" Sep 10 23:36:23.695953 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688099 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 23:36:23.695953 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688104 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 23:36:23.695953 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688110 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 23:36:23.695953 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688118 2578 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 23:36:23.695953 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688123 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 23:36:23.695953 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688127 2578 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 23:36:23.695953 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688132 2578 flags.go:64] FLAG: --logging-format="text" Sep 10 23:36:23.695953 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688137 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 23:36:23.695953 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688143 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 23:36:23.695953 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688147 2578 flags.go:64] FLAG: --manifest-url="" Sep 10 23:36:23.695953 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688152 2578 flags.go:64] FLAG: --manifest-url-header="" Sep 10 23:36:23.695953 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688158 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 23:36:23.695953 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688163 2578 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 23:36:23.695953 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688170 2578 flags.go:64] FLAG: --max-pods="110" Sep 10 23:36:23.695953 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688175 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 23:36:23.695953 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688180 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 23:36:23.695953 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688185 2578 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 23:36:23.695953 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688189 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 23:36:23.696911 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688195 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 23:36:23.696911 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688199 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 23:36:23.696911 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688204 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 23:36:23.696911 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688215 2578 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 23:36:23.696911 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688220 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 23:36:23.696911 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688224 2578 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 23:36:23.696911 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688229 2578 flags.go:64] FLAG: --pod-cidr="" Sep 10 23:36:23.696911 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688234 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 10 23:36:23.696911 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688242 2578 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 23:36:23.696911 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688247 2578 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 23:36:23.696911 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688253 2578 flags.go:64] FLAG: --pods-per-core="0" Sep 10 23:36:23.696911 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688258 2578 flags.go:64] FLAG: --port="10250" Sep 10 23:36:23.696911 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688265 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 23:36:23.696911 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688270 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-069b8471b632496d0" Sep 10 23:36:23.696911 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688276 2578 flags.go:64] FLAG: --qos-reserved="" Sep 10 23:36:23.696911 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688281 2578 flags.go:64] FLAG: --read-only-port="10255" Sep 10 23:36:23.696911 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688286 2578 flags.go:64] FLAG: --register-node="true" Sep 10 23:36:23.696911 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688291 2578 flags.go:64] FLAG: --register-with-taints="" Sep 10 23:36:23.696911 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688298 2578 flags.go:64] FLAG: --registry-burst="10" Sep 10 23:36:23.696911 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688302 2578 flags.go:64] FLAG: --registry-qps="5" Sep 10 23:36:23.696911 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688307 2578 flags.go:64] FLAG: --reserved-cpus="" Sep 10 23:36:23.696911 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688312 2578 flags.go:64] FLAG: --reserved-memory="" Sep 10 23:36:23.696911 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688318 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 23:36:23.696911 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688323 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 23:36:23.696911 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688328 2578 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 23:36:23.697954 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688333 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 23:36:23.697954 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688337 2578 flags.go:64] FLAG: --runonce="false" Sep 10 23:36:23.697954 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688342 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 23:36:23.697954 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688347 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 23:36:23.697954 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688352 2578 flags.go:64] FLAG: --seccomp-default="false" Sep 10 23:36:23.697954 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688356 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 23:36:23.697954 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688361 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 23:36:23.697954 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688366 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 23:36:23.697954 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688388 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 23:36:23.697954 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688393 2578 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 23:36:23.697954 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688397 2578 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 23:36:23.697954 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688402 2578 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 23:36:23.697954 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688406 2578 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 23:36:23.697954 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688410 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 23:36:23.697954 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688416 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 23:36:23.697954 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688420 2578 flags.go:64] FLAG: --system-cgroups="" Sep 10 23:36:23.697954 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688425 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 23:36:23.697954 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688434 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 23:36:23.697954 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688438 2578 flags.go:64] FLAG: --tls-cert-file="" Sep 10 23:36:23.697954 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688443 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 23:36:23.697954 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688450 2578 flags.go:64] FLAG: --tls-min-version="" Sep 10 23:36:23.697954 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688455 2578 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 23:36:23.697954 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688460 2578 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 23:36:23.697954 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688464 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 23:36:23.697954 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688469 2578 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 23:36:23.699050 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688478 2578 flags.go:64] FLAG: --v="2" Sep 10 23:36:23.699050 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688485 2578 flags.go:64] FLAG: --version="false" Sep 10 23:36:23.699050 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688491 2578 flags.go:64] FLAG: --vmodule="" Sep 10 23:36:23.699050 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688497 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 23:36:23.699050 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.688503 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 23:36:23.699050 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688679 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 23:36:23.699050 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688688 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 23:36:23.699050 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688693 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 23:36:23.699050 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688698 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 23:36:23.699050 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688702 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 23:36:23.699050 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688709 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 23:36:23.699050 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688714 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 23:36:23.699050 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688718 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 23:36:23.699050 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688723 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 23:36:23.699050 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688727 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 23:36:23.699050 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688731 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 23:36:23.699050 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688735 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 23:36:23.699050 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688740 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 23:36:23.699050 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688744 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 23:36:23.699050 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688748 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 23:36:23.699918 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688752 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 23:36:23.699918 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688758 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 23:36:23.699918 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688765 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 23:36:23.699918 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688770 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 23:36:23.699918 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688774 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 23:36:23.699918 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688778 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 23:36:23.699918 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688782 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 23:36:23.699918 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688787 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 23:36:23.699918 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688791 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 23:36:23.699918 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688795 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 23:36:23.699918 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688799 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 23:36:23.699918 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688803 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 23:36:23.699918 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688808 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 23:36:23.699918 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688813 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 23:36:23.699918 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688817 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 23:36:23.699918 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688824 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 23:36:23.699918 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688829 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 23:36:23.699918 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688833 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 23:36:23.699918 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688837 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 23:36:23.699918 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688841 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 23:36:23.700873 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688845 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 23:36:23.700873 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688850 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 23:36:23.700873 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688854 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 23:36:23.700873 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688858 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 23:36:23.700873 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688862 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 23:36:23.700873 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688869 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 23:36:23.700873 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688873 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 23:36:23.700873 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688877 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 23:36:23.700873 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688881 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 23:36:23.700873 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688885 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 23:36:23.700873 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688890 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 23:36:23.700873 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688894 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 23:36:23.700873 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688898 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 23:36:23.700873 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688902 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 23:36:23.700873 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688906 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 23:36:23.700873 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688910 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 23:36:23.700873 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688914 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 23:36:23.700873 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688918 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 23:36:23.700873 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688922 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 23:36:23.700873 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688925 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 23:36:23.701627 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688930 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 23:36:23.701627 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688934 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 23:36:23.701627 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688938 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 23:36:23.701627 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688942 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 23:36:23.701627 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688946 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 23:36:23.701627 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688951 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 23:36:23.701627 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688956 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 23:36:23.701627 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688963 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 23:36:23.701627 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688967 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 23:36:23.701627 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688971 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 10 23:36:23.701627 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688975 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 23:36:23.701627 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688979 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 23:36:23.701627 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688983 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 23:36:23.701627 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688987 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 23:36:23.701627 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688992 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 23:36:23.701627 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.688996 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 23:36:23.701627 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689000 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 23:36:23.701627 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689006 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 23:36:23.701627 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689010 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 23:36:23.702178 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689014 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 23:36:23.702178 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689019 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 23:36:23.702178 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689023 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 23:36:23.702178 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689027 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 23:36:23.702178 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689032 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 23:36:23.702178 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689037 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 23:36:23.702178 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689041 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 23:36:23.702178 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689045 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 23:36:23.702178 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689049 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 23:36:23.702178 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689054 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 23:36:23.702178 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689059 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 23:36:23.702178 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689063 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 23:36:23.702178 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689067 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 23:36:23.702178 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689071 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 23:36:23.702178 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689075 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 23:36:23.702178 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689079 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 23:36:23.702178 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689084 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 23:36:23.702178 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689088 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 23:36:23.702178 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689092 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 23:36:23.702178 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689096 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 23:36:23.702909 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689103 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 23:36:23.702909 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689107 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 23:36:23.702909 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689111 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 23:36:23.702909 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689115 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 23:36:23.702909 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.689119 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 23:36:23.702909 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.689854 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 23:36:23.702909 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.699873 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 23:36:23.702909 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.699892 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 23:36:23.702909 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.699918 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 23:36:23.702909 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.699931 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 23:36:23.702909 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700014 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 23:36:23.702909 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700022 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 23:36:23.702909 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700027 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 23:36:23.702909 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700033 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 23:36:23.702909 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700037 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 23:36:23.703291 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700041 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 23:36:23.703291 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700046 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 23:36:23.703291 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700050 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 23:36:23.703291 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700054 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 23:36:23.703291 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700059 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 23:36:23.703291 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700065 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 23:36:23.703291 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700072 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 23:36:23.703291 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700077 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 10 23:36:23.703291 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700082 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 23:36:23.703291 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700087 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 23:36:23.703291 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700092 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 23:36:23.703291 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700096 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 23:36:23.703291 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700102 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 23:36:23.703291 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700106 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 23:36:23.703291 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700110 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 23:36:23.703291 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700114 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 23:36:23.703291 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700118 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 23:36:23.703291 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700123 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 23:36:23.703291 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700127 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 23:36:23.703291 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700132 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 23:36:23.704045 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700136 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 23:36:23.704045 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700140 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 23:36:23.704045 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700145 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 23:36:23.704045 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700149 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 23:36:23.704045 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700153 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 23:36:23.704045 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700157 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 23:36:23.704045 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700162 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 23:36:23.704045 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700168 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 23:36:23.704045 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700175 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 23:36:23.704045 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700181 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 23:36:23.704045 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700186 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 23:36:23.704045 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700190 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 23:36:23.704045 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700194 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 23:36:23.704045 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700198 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 23:36:23.704045 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700202 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 23:36:23.704045 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700206 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 23:36:23.704045 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700211 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 23:36:23.704045 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700215 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 23:36:23.704045 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700219 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 23:36:23.704789 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700224 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 23:36:23.704789 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700228 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 23:36:23.704789 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700233 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 23:36:23.704789 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700237 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 23:36:23.704789 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700242 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 23:36:23.704789 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700246 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 23:36:23.704789 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700251 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 23:36:23.704789 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700255 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 23:36:23.704789 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700259 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 23:36:23.704789 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700264 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 23:36:23.704789 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700268 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 23:36:23.704789 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700272 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 23:36:23.704789 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700276 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 23:36:23.704789 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700283 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 23:36:23.704789 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700289 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 23:36:23.704789 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700294 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 23:36:23.704789 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700299 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 23:36:23.704789 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700304 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 23:36:23.704789 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700309 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 23:36:23.704789 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700313 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 23:36:23.705350 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700318 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 23:36:23.705350 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700323 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 23:36:23.705350 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700327 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 23:36:23.705350 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700331 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 23:36:23.705350 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700335 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 23:36:23.705350 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700339 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 23:36:23.705350 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700344 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 23:36:23.705350 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700348 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 23:36:23.705350 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700353 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 23:36:23.705350 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700357 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 23:36:23.705350 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700361 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 23:36:23.705350 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700365 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 23:36:23.705350 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700386 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 23:36:23.705350 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700390 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 23:36:23.705350 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700394 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 23:36:23.705350 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700399 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 23:36:23.705350 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700404 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 23:36:23.705350 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700408 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 23:36:23.705350 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700412 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 23:36:23.705350 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700416 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 23:36:23.705963 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700420 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 23:36:23.705963 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700425 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 23:36:23.705963 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700429 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 23:36:23.705963 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700433 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 23:36:23.705963 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700438 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 23:36:23.705963 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700442 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 23:36:23.705963 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700446 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 23:36:23.705963 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700450 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 23:36:23.705963 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700454 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 23:36:23.705963 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700458 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 23:36:23.705963 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700462 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 23:36:23.705963 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700466 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 23:36:23.705963 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700471 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 23:36:23.705963 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700475 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 23:36:23.705963 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700479 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 23:36:23.706338 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.700487 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 23:36:23.706338 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700680 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 23:36:23.706338 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700690 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 23:36:23.706338 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700694 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 23:36:23.706338 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700700 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 23:36:23.706338 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700705 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 23:36:23.706338 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700710 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 23:36:23.706338 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700715 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 23:36:23.706338 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700720 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 23:36:23.706338 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700725 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 23:36:23.706338 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700730 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 23:36:23.706338 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700734 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 23:36:23.706338 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700739 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 23:36:23.706338 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700744 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 23:36:23.706757 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700748 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 23:36:23.706757 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700753 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 23:36:23.706757 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700757 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 23:36:23.706757 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700761 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 23:36:23.706757 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700767 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 23:36:23.706757 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700772 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 23:36:23.706757 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700776 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 23:36:23.706757 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700781 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 23:36:23.706757 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700785 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 23:36:23.706757 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700789 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 23:36:23.706757 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700793 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 23:36:23.706757 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700797 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 23:36:23.706757 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700802 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 23:36:23.706757 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700806 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 23:36:23.706757 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700811 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 23:36:23.706757 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700816 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 23:36:23.706757 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700820 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 23:36:23.706757 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700824 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 23:36:23.706757 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700829 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 23:36:23.706757 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700833 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 23:36:23.707244 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700837 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 23:36:23.707244 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700841 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 23:36:23.707244 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700845 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 23:36:23.707244 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700850 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 23:36:23.707244 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700857 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 23:36:23.707244 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700863 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 23:36:23.707244 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700868 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 23:36:23.707244 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700872 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 23:36:23.707244 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700877 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 23:36:23.707244 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700881 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 23:36:23.707244 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700885 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 23:36:23.707244 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700889 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 23:36:23.707244 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700895 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 23:36:23.707244 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700900 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 23:36:23.707244 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700904 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 23:36:23.707244 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700909 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 23:36:23.707244 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700913 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 23:36:23.707244 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700918 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 23:36:23.707244 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700922 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 23:36:23.707877 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700926 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 23:36:23.707877 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700930 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 23:36:23.707877 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700934 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 23:36:23.707877 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700938 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 23:36:23.707877 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700942 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 23:36:23.707877 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700946 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 23:36:23.707877 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700950 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 23:36:23.707877 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700954 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 23:36:23.707877 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700959 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 23:36:23.707877 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700964 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 23:36:23.707877 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700968 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 23:36:23.707877 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700972 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 23:36:23.707877 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700976 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 23:36:23.707877 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700980 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 23:36:23.707877 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700984 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 23:36:23.707877 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700989 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 23:36:23.707877 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700993 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 23:36:23.707877 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.700998 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 23:36:23.707877 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701002 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 23:36:23.707877 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701006 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 23:36:23.708394 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701010 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 23:36:23.708394 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701014 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 23:36:23.708394 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701019 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 23:36:23.708394 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701023 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 23:36:23.708394 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701027 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 23:36:23.708394 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701031 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 23:36:23.708394 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701037 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 23:36:23.708394 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701041 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 23:36:23.708394 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701046 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 10 23:36:23.708394 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701049 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 23:36:23.708394 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701053 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 23:36:23.708394 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701058 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 23:36:23.708394 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701062 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 23:36:23.708394 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701066 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 23:36:23.708394 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701071 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 23:36:23.708394 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701075 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 23:36:23.708394 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701079 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 23:36:23.708394 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701083 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 23:36:23.708394 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701087 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 23:36:23.708394 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701091 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 23:36:23.708879 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701095 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 23:36:23.708879 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701099 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 23:36:23.708879 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701103 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 23:36:23.708879 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701107 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 23:36:23.708879 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701112 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 23:36:23.708879 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701116 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 23:36:23.708879 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:23.701121 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 23:36:23.708879 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.701129 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 23:36:23.708879 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.701284 2578 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 23:36:23.708879 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.705345 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 23:36:23.708879 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.706459 2578 server.go:1019] "Starting client certificate rotation" Sep 10 23:36:23.708879 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.706560 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 23:36:23.708879 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.707256 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 23:36:23.709505 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.709490 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 23:36:23.710637 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.710619 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 23:36:23.723103 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.723087 2578 log.go:25] "Validated CRI v1 runtime API" Sep 10 23:36:23.728464 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.728448 2578 log.go:25] "Validated CRI v1 image API" Sep 10 23:36:23.730018 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.730003 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 23:36:23.733166 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.733132 2578 fs.go:135] Filesystem UUIDs: map[6824605e-2ffc-4ec3-befb-7326fc0aae96:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 85f46662-1751-4177-b7e8-323f5be93f97:/dev/nvme0n1p4] Sep 10 23:36:23.733230 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.733165 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 23:36:23.736577 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.736558 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 23:36:23.738809 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.738703 2578 manager.go:216] Machine: {Timestamp:2026-09-10 23:36:23.736935747 +0000 UTC m=+0.367973961 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099018 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29e44aa7da2ee2d449e2b7201b65ff SystemUUID:ec29e44a-a7da-2ee2-d449-e2b7201b65ff BootID:183235b9-d77b-470e-a705-fc5d0fd6f51a Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:29:45:e2:d6:e9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:29:45:e2:d6:e9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:41:7e:4d:cb:a0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 23:36:23.738809 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.738809 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 23:36:23.738933 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.738921 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 23:36:23.742114 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.742090 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 23:36:23.742257 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.742117 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-142.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 23:36:23.742991 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.742981 2578 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 23:36:23.743021 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.742994 2578 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 23:36:23.743021 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.743002 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 23:36:23.743811 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.743800 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 23:36:23.744788 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.744778 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 23:36:23.748876 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.748863 2578 state_mem.go:36] "Initialized new in-memory state store" Sep 10 23:36:23.749001 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.748991 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 23:36:23.749031 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.749022 2578 kubelet.go:486] "Attempting to sync node with API server" Sep 10 23:36:23.749058 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.749040 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 23:36:23.749093 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.749060 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 23:36:23.749093 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.749069 2578 kubelet.go:398] "Adding apiserver pod source" Sep 10 23:36:23.749093 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.749078 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 23:36:23.749474 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.749463 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 23:36:23.749515 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.749482 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 23:36:23.751728 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.751713 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 23:36:23.754531 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.754517 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 23:36:23.754598 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.754547 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 23:36:23.756461 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.756450 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 23:36:23.756496 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.756465 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 23:36:23.756496 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.756470 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 23:36:23.756496 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.756476 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 23:36:23.756496 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.756484 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 23:36:23.756496 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.756490 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 23:36:23.756629 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.756504 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 23:36:23.756629 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.756510 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 23:36:23.756629 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.756516 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 23:36:23.756629 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.756521 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 23:36:23.756629 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.756529 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 23:36:23.756629 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.756535 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 23:36:23.757332 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.757323 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 23:36:23.757365 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.757333 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 23:36:23.760292 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.760277 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-142.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 23:36:23.760365 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:23.760301 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 23:36:23.760485 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:23.760466 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-142.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 23:36:23.761225 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.761212 2578 server.go:1268] "Started kubelet" Sep 10 23:36:23.761305 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.761282 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 23:36:23.761943 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.761905 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 23:36:23.761999 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.761958 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 23:36:23.763297 ip-10-0-139-142 systemd[1]: Started Kubernetes Kubelet. Sep 10 23:36:23.763505 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.763347 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 23:36:23.768414 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:23.768386 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 23:36:23.768520 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.768489 2578 server.go:310] "Adding debug handlers to kubelet server" Sep 10 23:36:23.771488 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:23.770316 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-142.ec2.internal.18d419dbcaa39f8f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-142.ec2.internal,UID:ip-10-0-139-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-142.ec2.internal,},FirstTimestamp:2026-09-10 23:36:23.761182607 +0000 UTC m=+0.392220821,LastTimestamp:2026-09-10 23:36:23.761182607 +0000 UTC m=+0.392220821,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-142.ec2.internal,}" Sep 10 23:36:23.772968 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.772951 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 23:36:23.774176 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.774162 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 23:36:23.774289 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:23.774270 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-142.ec2.internal\" not found" Sep 10 23:36:23.774995 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.774941 2578 factory.go:55] Registering systemd factory Sep 10 23:36:23.774995 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.774964 2578 factory.go:223] Registration of the systemd container factory successfully Sep 10 23:36:23.775197 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.775182 2578 factory.go:153] Registering CRI-O factory Sep 10 23:36:23.775274 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.775201 2578 factory.go:223] Registration of the crio container factory successfully Sep 10 23:36:23.775274 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.775262 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 23:36:23.775349 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.775273 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 23:36:23.775349 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.775276 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 23:36:23.775349 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.775295 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 23:36:23.775349 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.775280 2578 factory.go:103] Registering Raw factory Sep 10 23:36:23.775349 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.775347 2578 manager.go:1196] Started watching for new ooms in manager Sep 10 23:36:23.775768 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.775754 2578 manager.go:349] Starting recovery of all containers Sep 10 23:36:23.776402 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.776383 2578 reconstruct.go:97] "Volume reconstruction finished" Sep 10 23:36:23.776402 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.776399 2578 reconciler.go:29] "Reconciler: start to sync state" Sep 10 23:36:23.777543 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:23.777520 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 23:36:23.777543 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:23.777520 2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-142.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 10 23:36:23.781966 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.781941 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mv4rz" Sep 10 23:36:23.788719 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.788699 2578 manager.go:354] Recovery completed Sep 10 23:36:23.789642 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.789623 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mv4rz" Sep 10 23:36:23.793655 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.793635 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 23:36:23.795525 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.795503 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientMemory" Sep 10 23:36:23.795602 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.795541 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 23:36:23.795602 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.795555 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientPID" Sep 10 23:36:23.796240 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.796223 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 23:36:23.796325 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.796224 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 23:36:23.796325 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.796294 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 23:36:23.796325 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.796314 2578 state_mem.go:36] "Initialized new in-memory state store" Sep 10 23:36:23.796983 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.796964 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientMemory" Sep 10 23:36:23.797073 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.796991 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 23:36:23.797073 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.797001 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientPID" Sep 10 23:36:23.797516 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:23.797420 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-142.ec2.internal.18d419dbccafa316 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-142.ec2.internal,UID:ip-10-0-139-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-142.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-142.ec2.internal,},FirstTimestamp:2026-09-10 23:36:23.795524374 +0000 UTC m=+0.426562589,LastTimestamp:2026-09-10 23:36:23.795524374 +0000 UTC m=+0.426562589,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-142.ec2.internal,}" Sep 10 23:36:23.798321 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.798301 2578 policy_none.go:49] "None policy: Start" Sep 10 23:36:23.798321 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.798318 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 23:36:23.798435 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.798329 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 23:36:23.800277 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.800266 2578 policy_none.go:47] "Start" Sep 10 23:36:23.833983 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.833967 2578 manager.go:337] "Starting Device Plugin manager" Sep 10 23:36:23.850981 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:23.834006 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 23:36:23.850981 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.834015 2578 server.go:87] "Starting device plugin registration server" Sep 10 23:36:23.850981 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.834271 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 23:36:23.850981 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.834284 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 23:36:23.850981 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.834402 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 23:36:23.850981 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.834486 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 23:36:23.850981 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.834496 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 23:36:23.850981 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:23.834996 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 23:36:23.850981 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:23.835042 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-142.ec2.internal\" not found" Sep 10 23:36:23.873068 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.873041 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 23:36:23.874160 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.874143 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 23:36:23.874223 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.874207 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 23:36:23.874268 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.874244 2578 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 23:36:23.874310 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:23.874294 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 23:36:23.876048 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.876029 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 23:36:23.935190 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.935112 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 23:36:23.936228 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.936209 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientMemory" Sep 10 23:36:23.936305 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.936245 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 23:36:23.936305 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.936258 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientPID" Sep 10 23:36:23.936305 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.936291 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-142.ec2.internal" Sep 10 23:36:23.942673 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.942656 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-142.ec2.internal" Sep 10 23:36:23.942725 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:23.942681 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-142.ec2.internal\": node \"ip-10-0-139-142.ec2.internal\" not found" Sep 10 23:36:23.969114 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:23.969088 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-142.ec2.internal\" not found" Sep 10 23:36:23.975139 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.975108 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-142.ec2.internal"] Sep 10 23:36:23.975249 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.975189 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 23:36:23.976238 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.976225 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientMemory" Sep 10 23:36:23.976289 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.976255 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 23:36:23.976289 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.976268 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientPID" Sep 10 23:36:23.977353 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.977337 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6965b1dacc260bd4d7bb7a14199bc848-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal\" (UID: \"6965b1dacc260bd4d7bb7a14199bc848\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal" Sep 10 23:36:23.977416 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.977361 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6965b1dacc260bd4d7bb7a14199bc848-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal\" (UID: \"6965b1dacc260bd4d7bb7a14199bc848\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal" Sep 10 23:36:23.978520 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.978506 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 23:36:23.978598 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.978586 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal" Sep 10 23:36:23.978652 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.978612 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 23:36:23.979308 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.979289 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientMemory" Sep 10 23:36:23.979432 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.979322 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 23:36:23.979432 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.979333 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientPID" Sep 10 23:36:23.979432 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.979333 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientMemory" Sep 10 23:36:23.979432 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.979363 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 23:36:23.979432 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.979387 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientPID" Sep 10 23:36:23.981508 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.981494 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-142.ec2.internal" Sep 10 23:36:23.981580 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.981519 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 23:36:23.982250 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.982228 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientMemory" Sep 10 23:36:23.982296 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.982264 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 23:36:23.982296 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:23.982280 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeHasSufficientPID" Sep 10 23:36:24.001057 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:24.001036 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-142.ec2.internal\" not found" node="ip-10-0-139-142.ec2.internal" Sep 10 23:36:24.005619 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:24.005602 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-142.ec2.internal\" not found" node="ip-10-0-139-142.ec2.internal" Sep 10 23:36:24.069528 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:24.069501 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-142.ec2.internal\" not found" Sep 10 23:36:24.078242 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:24.078219 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6965b1dacc260bd4d7bb7a14199bc848-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal\" (UID: \"6965b1dacc260bd4d7bb7a14199bc848\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal" Sep 10 23:36:24.078318 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:24.078252 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6965b1dacc260bd4d7bb7a14199bc848-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal\" (UID: \"6965b1dacc260bd4d7bb7a14199bc848\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal" Sep 10 23:36:24.078318 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:24.078276 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bff3cfe4b5e3fbd1418d844a741d3b63-config\") pod \"kube-apiserver-proxy-ip-10-0-139-142.ec2.internal\" (UID: \"bff3cfe4b5e3fbd1418d844a741d3b63\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-142.ec2.internal" Sep 10 23:36:24.078443 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:24.078327 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6965b1dacc260bd4d7bb7a14199bc848-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal\" (UID: \"6965b1dacc260bd4d7bb7a14199bc848\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal" Sep 10 23:36:24.078443 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:24.078362 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6965b1dacc260bd4d7bb7a14199bc848-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal\" (UID: \"6965b1dacc260bd4d7bb7a14199bc848\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal" Sep 10 23:36:24.170569 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:24.170535 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-142.ec2.internal\" not found" Sep 10 23:36:24.178922 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:24.178901 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bff3cfe4b5e3fbd1418d844a741d3b63-config\") pod \"kube-apiserver-proxy-ip-10-0-139-142.ec2.internal\" (UID: \"bff3cfe4b5e3fbd1418d844a741d3b63\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-142.ec2.internal" Sep 10 23:36:24.178998 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:24.178955 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bff3cfe4b5e3fbd1418d844a741d3b63-config\") pod \"kube-apiserver-proxy-ip-10-0-139-142.ec2.internal\" (UID: \"bff3cfe4b5e3fbd1418d844a741d3b63\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-142.ec2.internal" Sep 10 23:36:24.271433 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:24.271351 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-142.ec2.internal\" not found" Sep 10 23:36:24.303874 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:24.303851 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal" Sep 10 23:36:24.308907 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:24.308603 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-142.ec2.internal" Sep 10 23:36:25.535712 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:25.535691 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-142.ec2.internal\" not found" Sep 10 23:36:25.636273 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:25.636241 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-142.ec2.internal\" not found" Sep 10 23:36:25.736876 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:25.736792 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-142.ec2.internal\" not found" Sep 10 23:36:25.837357 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:25.837328 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-142.ec2.internal\" not found" Sep 10 23:36:25.870795 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:25.870769 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 23:36:25.871245 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:25.870967 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 23:36:25.934733 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:25.934707 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 23:36:25.937942 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:25.937917 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-142.ec2.internal\" not found" Sep 10 23:36:25.958879 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:25.958846 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 23:31:24 +0000 UTC" deadline="2028-06-09 20:10:43.758076558 +0000 UTC" Sep 10 23:36:25.958879 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:25.958877 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15308h34m17.799204108s" Sep 10 23:36:26.038198 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:26.038172 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-142.ec2.internal\" not found" Sep 10 23:36:26.042365 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:26.042330 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff3cfe4b5e3fbd1418d844a741d3b63.slice/crio-e4e17c820c947dd02e7766bd07a5530c6a4c97f64e45854f950e28117291fec4 WatchSource:0}: Error finding container e4e17c820c947dd02e7766bd07a5530c6a4c97f64e45854f950e28117291fec4: Status 404 returned error can't find the container with id e4e17c820c947dd02e7766bd07a5530c6a4c97f64e45854f950e28117291fec4 Sep 10 23:36:26.042584 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:26.042561 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6965b1dacc260bd4d7bb7a14199bc848.slice/crio-91b3b90715bea303cad8e0cc057f44ca8fd747cbaf1cb29d4f39c94be0f36524 WatchSource:0}: Error finding container 91b3b90715bea303cad8e0cc057f44ca8fd747cbaf1cb29d4f39c94be0f36524: Status 404 returned error can't find the container with id 91b3b90715bea303cad8e0cc057f44ca8fd747cbaf1cb29d4f39c94be0f36524 Sep 10 23:36:26.045798 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.045782 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 23:36:26.117093 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.117061 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 23:36:26.131451 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.131429 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 23:36:26.138179 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.138157 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal" Sep 10 23:36:26.151245 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.151226 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 23:36:26.152251 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.152240 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-142.ec2.internal" Sep 10 23:36:26.161055 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.161039 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 23:36:26.913219 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.913176 2578 apiserver.go:52] "Watching apiserver" Sep 10 23:36:26.920012 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.919957 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 23:36:26.920512 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.920489 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-whckt","openshift-network-diagnostics/network-check-target-fb4qx","openshift-ovn-kubernetes/ovnkube-node-8srbc","kube-system/global-pull-secret-syncer-zzfhc","kube-system/konnectivity-agent-kr8fb","openshift-dns/node-resolver-dm6sw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal","openshift-multus/multus-v77rf","openshift-multus/network-metrics-daemon-fmvxw","openshift-network-operator/iptables-alerter-jfzt2","kube-system/kube-apiserver-proxy-ip-10-0-139-142.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56","openshift-cluster-node-tuning-operator/tuned-5pgnj","openshift-image-registry/node-ca-f7fjr"] Sep 10 23:36:26.923227 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.923197 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jfzt2" Sep 10 23:36:26.925436 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.925403 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:26.925679 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.925658 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 23:36:26.925949 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.925919 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kfrhm\"" Sep 10 23:36:26.926026 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.925990 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 23:36:26.926216 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.926197 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 23:36:26.927800 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.927291 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 23:36:26.927800 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.927652 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 23:36:26.927800 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.927743 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:26.928681 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.928518 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k77kt\"" Sep 10 23:36:26.930202 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.930182 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:26.930292 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:26.930269 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzfhc" podUID="9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7" Sep 10 23:36:26.930904 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.930746 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 23:36:26.930904 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.930754 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 23:36:26.930904 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.930765 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 23:36:26.931116 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.931020 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 23:36:26.931357 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.931337 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ktc65\"" Sep 10 23:36:26.931535 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.931522 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 23:36:26.931798 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.931783 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 23:36:26.932403 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.932385 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kr8fb" Sep 10 23:36:26.934692 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.934361 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 23:36:26.934692 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.934555 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 23:36:26.934692 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.934640 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-kt4jq\"" Sep 10 23:36:26.935477 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.934964 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dm6sw" Sep 10 23:36:26.936981 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.936961 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 23:36:26.937067 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.937004 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 23:36:26.937261 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.937244 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 23:36:26.937716 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.937696 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:26.937822 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:26.937767 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmvxw" podUID="a06f8008-28ac-4cb9-8310-f5c42b169d10" Sep 10 23:36:26.938007 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.937987 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6lptc\"" Sep 10 23:36:26.940542 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.940026 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f7fjr" Sep 10 23:36:26.942988 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.942486 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:36:26.942988 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:26.942541 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fb4qx" podUID="c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d" Sep 10 23:36:26.942988 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.942769 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 23:36:26.942988 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.942809 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hf66z\"" Sep 10 23:36:26.943171 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.943043 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 23:36:26.943439 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.943261 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 23:36:26.945031 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.944688 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-whckt" Sep 10 23:36:26.948436 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.947320 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 23:36:26.948436 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.947540 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 23:36:26.948436 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.947650 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 23:36:26.948436 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.947718 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 23:36:26.948436 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.947857 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-28p27\"" Sep 10 23:36:26.948436 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.947963 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v77rf" Sep 10 23:36:26.951709 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.951678 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-env-overrides\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:26.951810 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.951726 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq45v\" (UniqueName: \"kubernetes.io/projected/7be68ac6-c59b-4afa-9974-a39f8067c1f7-kube-api-access-pq45v\") pod \"node-ca-f7fjr\" (UID: \"7be68ac6-c59b-4afa-9974-a39f8067c1f7\") " pod="openshift-image-registry/node-ca-f7fjr" Sep 10 23:36:26.951810 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.951754 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-kubelet-config\") pod \"global-pull-secret-syncer-zzfhc\" (UID: \"9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7\") " pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:26.951810 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.951794 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1da41227-eab5-43fa-a97c-69f7d935cc76-hosts-file\") pod \"node-resolver-dm6sw\" (UID: \"1da41227-eab5-43fa-a97c-69f7d935cc76\") " pod="openshift-dns/node-resolver-dm6sw" Sep 10 23:36:26.951983 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.951834 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1da41227-eab5-43fa-a97c-69f7d935cc76-tmp-dir\") pod \"node-resolver-dm6sw\" (UID: \"1da41227-eab5-43fa-a97c-69f7d935cc76\") " pod="openshift-dns/node-resolver-dm6sw" Sep 10 23:36:26.951983 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.951863 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-etc-systemd\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:26.951983 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.951874 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 23:36:26.951983 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.951894 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-var-lib-kubelet\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:26.951983 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.951918 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-host-slash\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:26.952219 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.951983 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-run-ovn\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:26.952219 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952013 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-ovn-node-metrics-cert\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:26.952219 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952043 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4b358463-cfe2-49ec-91ea-64ae0856ccd7-agent-certs\") pod \"konnectivity-agent-kr8fb\" (UID: \"4b358463-cfe2-49ec-91ea-64ae0856ccd7\") " pod="kube-system/konnectivity-agent-kr8fb" Sep 10 23:36:26.952219 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952079 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-original-pull-secret\") pod \"global-pull-secret-syncer-zzfhc\" (UID: \"9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7\") " pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:26.952219 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952104 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a055ce8-9e13-4aef-9d72-d14e198166a8-host-slash\") pod \"iptables-alerter-jfzt2\" (UID: \"3a055ce8-9e13-4aef-9d72-d14e198166a8\") " pod="openshift-network-operator/iptables-alerter-jfzt2" Sep 10 23:36:26.952219 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952135 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-etc-sysctl-d\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:26.952219 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952161 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-host\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:26.952219 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952192 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8df0f6be-ce76-4263-adfb-b1531272132e-etc-tuned\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:26.952219 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952219 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-host-run-netns\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:26.952636 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952248 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" Sep 10 23:36:26.952636 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952253 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-ovnkube-script-lib\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:26.952636 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952279 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7be68ac6-c59b-4afa-9974-a39f8067c1f7-host\") pod \"node-ca-f7fjr\" (UID: \"7be68ac6-c59b-4afa-9974-a39f8067c1f7\") " pod="openshift-image-registry/node-ca-f7fjr" Sep 10 23:36:26.952636 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952314 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7be68ac6-c59b-4afa-9974-a39f8067c1f7-serviceca\") pod \"node-ca-f7fjr\" (UID: \"7be68ac6-c59b-4afa-9974-a39f8067c1f7\") " pod="openshift-image-registry/node-ca-f7fjr" Sep 10 23:36:26.952636 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952345 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3a055ce8-9e13-4aef-9d72-d14e198166a8-iptables-alerter-script\") pod \"iptables-alerter-jfzt2\" (UID: \"3a055ce8-9e13-4aef-9d72-d14e198166a8\") " pod="openshift-network-operator/iptables-alerter-jfzt2" Sep 10 23:36:26.952636 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952373 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-sys\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:26.952636 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952397 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8df0f6be-ce76-4263-adfb-b1531272132e-tmp\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:26.952636 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952428 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-host-kubelet\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:26.952636 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952458 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-systemd-units\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:26.952636 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952487 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-etc-openvswitch\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:26.952636 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952515 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-node-log\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:26.952636 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952571 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-etc-sysconfig\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:26.952636 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952603 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-var-lib-openvswitch\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:26.952636 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952634 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-host-run-ovn-kubernetes\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:26.953280 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952664 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-host-cni-netd\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:26.953280 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952695 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a06f8008-28ac-4cb9-8310-f5c42b169d10-metrics-certs\") pod \"network-metrics-daemon-fmvxw\" (UID: \"a06f8008-28ac-4cb9-8310-f5c42b169d10\") " pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:26.953280 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952726 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-dbus\") pod \"global-pull-secret-syncer-zzfhc\" (UID: \"9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7\") " pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:26.953280 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952759 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-lib-modules\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:26.953280 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952805 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7czj\" (UniqueName: \"kubernetes.io/projected/8df0f6be-ce76-4263-adfb-b1531272132e-kube-api-access-b7czj\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:26.953280 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952837 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:26.953280 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952897 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tkc7\" (UniqueName: \"kubernetes.io/projected/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-kube-api-access-6tkc7\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:26.953280 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.952957 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kv9s\" (UniqueName: \"kubernetes.io/projected/1da41227-eab5-43fa-a97c-69f7d935cc76-kube-api-access-2kv9s\") pod \"node-resolver-dm6sw\" (UID: \"1da41227-eab5-43fa-a97c-69f7d935cc76\") " pod="openshift-dns/node-resolver-dm6sw" Sep 10 23:36:26.953280 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.953003 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7lbv\" (UniqueName: \"kubernetes.io/projected/3a055ce8-9e13-4aef-9d72-d14e198166a8-kube-api-access-h7lbv\") pod \"iptables-alerter-jfzt2\" (UID: \"3a055ce8-9e13-4aef-9d72-d14e198166a8\") " pod="openshift-network-operator/iptables-alerter-jfzt2" Sep 10 23:36:26.953280 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.953036 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-etc-modprobe-d\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:26.953280 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.953068 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-etc-sysctl-conf\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:26.953280 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.953105 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-run\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:26.953280 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.953136 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-run-systemd\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:26.953280 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.953173 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-host-cni-bin\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:26.953988 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.953296 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-ovnkube-config\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:26.953988 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.953327 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4b358463-cfe2-49ec-91ea-64ae0856ccd7-konnectivity-ca\") pod \"konnectivity-agent-kr8fb\" (UID: \"4b358463-cfe2-49ec-91ea-64ae0856ccd7\") " pod="kube-system/konnectivity-agent-kr8fb" Sep 10 23:36:26.953988 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.953359 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9lpp\" (UniqueName: \"kubernetes.io/projected/a06f8008-28ac-4cb9-8310-f5c42b169d10-kube-api-access-m9lpp\") pod \"network-metrics-daemon-fmvxw\" (UID: \"a06f8008-28ac-4cb9-8310-f5c42b169d10\") " pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:26.953988 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.953388 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-etc-kubernetes\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:26.953988 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.953414 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-run-openvswitch\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:26.953988 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.953445 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-log-socket\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:26.954647 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.954596 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 23:36:26.955313 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.954801 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 23:36:26.955313 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.954905 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 23:36:26.955313 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.955104 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 23:36:26.955313 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.955133 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dzgfc\"" Sep 10 23:36:26.957189 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.956412 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 23:36:26.957557 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.957532 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tg2vs\"" Sep 10 23:36:26.970906 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.970882 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-k9xlm" Sep 10 23:36:26.978642 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:26.978620 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-k9xlm" Sep 10 23:36:27.039997 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.039967 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 23:36:27.043924 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.043868 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-142.ec2.internal" event={"ID":"bff3cfe4b5e3fbd1418d844a741d3b63","Type":"ContainerStarted","Data":"e4e17c820c947dd02e7766bd07a5530c6a4c97f64e45854f950e28117291fec4"} Sep 10 23:36:27.045334 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.045312 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal" event={"ID":"6965b1dacc260bd4d7bb7a14199bc848","Type":"ContainerStarted","Data":"91b3b90715bea303cad8e0cc057f44ca8fd747cbaf1cb29d4f39c94be0f36524"} Sep 10 23:36:27.053830 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.053797 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-multus-socket-dir-parent\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.053972 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.053849 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-etc-sysconfig\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.053972 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.053879 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-var-lib-openvswitch\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.053972 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.053913 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-host-run-ovn-kubernetes\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.053972 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.053957 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-host-cni-netd\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.054166 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.053973 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-etc-sysconfig\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.054166 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.053978 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-var-lib-openvswitch\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.054166 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.053981 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a06f8008-28ac-4cb9-8310-f5c42b169d10-metrics-certs\") pod \"network-metrics-daemon-fmvxw\" (UID: \"a06f8008-28ac-4cb9-8310-f5c42b169d10\") " pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:27.054166 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054032 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-dbus\") pod \"global-pull-secret-syncer-zzfhc\" (UID: \"9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7\") " pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:27.054166 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054044 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-host-cni-netd\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.054166 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054063 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-system-cni-dir\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.054166 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:27.054095 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:27.054166 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054114 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-lib-modules\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.054166 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054123 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-host-run-ovn-kubernetes\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.054166 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054161 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.054715 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:27.054192 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a06f8008-28ac-4cb9-8310-f5c42b169d10-metrics-certs podName:a06f8008-28ac-4cb9-8310-f5c42b169d10 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:27.554152482 +0000 UTC m=+3.021626802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a06f8008-28ac-4cb9-8310-f5c42b169d10-metrics-certs") pod "network-metrics-daemon-fmvxw" (UID: "a06f8008-28ac-4cb9-8310-f5c42b169d10") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:27.054715 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054213 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.054715 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054232 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7lbv\" (UniqueName: \"kubernetes.io/projected/3a055ce8-9e13-4aef-9d72-d14e198166a8-kube-api-access-h7lbv\") pod \"iptables-alerter-jfzt2\" (UID: \"3a055ce8-9e13-4aef-9d72-d14e198166a8\") " pod="openshift-network-operator/iptables-alerter-jfzt2" Sep 10 23:36:27.054715 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054251 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-dbus\") pod \"global-pull-secret-syncer-zzfhc\" (UID: \"9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7\") " pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:27.054715 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054262 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4580695f-ec27-45ad-9670-4849d9354b05-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-whckt\" (UID: \"4580695f-ec27-45ad-9670-4849d9354b05\") " pod="openshift-multus/multus-additional-cni-plugins-whckt" Sep 10 23:36:27.054715 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054287 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-host-run-netns\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.054715 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054312 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-etc-modprobe-d\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.054715 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054313 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-lib-modules\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.054715 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054353 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-etc-sysctl-conf\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.054715 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054406 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-run\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.054715 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054409 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-etc-modprobe-d\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.054715 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054445 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-run-systemd\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.054715 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054459 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-run\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.054715 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054476 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4b358463-cfe2-49ec-91ea-64ae0856ccd7-konnectivity-ca\") pod \"konnectivity-agent-kr8fb\" (UID: \"4b358463-cfe2-49ec-91ea-64ae0856ccd7\") " pod="kube-system/konnectivity-agent-kr8fb" Sep 10 23:36:27.054715 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054500 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-run-systemd\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.054715 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054505 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9lpp\" (UniqueName: \"kubernetes.io/projected/a06f8008-28ac-4cb9-8310-f5c42b169d10-kube-api-access-m9lpp\") pod \"network-metrics-daemon-fmvxw\" (UID: \"a06f8008-28ac-4cb9-8310-f5c42b169d10\") " pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:27.055534 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054476 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-etc-sysctl-conf\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.055534 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054532 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1da41227-eab5-43fa-a97c-69f7d935cc76-hosts-file\") pod \"node-resolver-dm6sw\" (UID: \"1da41227-eab5-43fa-a97c-69f7d935cc76\") " pod="openshift-dns/node-resolver-dm6sw" Sep 10 23:36:27.055534 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054554 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1da41227-eab5-43fa-a97c-69f7d935cc76-tmp-dir\") pod \"node-resolver-dm6sw\" (UID: \"1da41227-eab5-43fa-a97c-69f7d935cc76\") " pod="openshift-dns/node-resolver-dm6sw" Sep 10 23:36:27.055534 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054577 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-env-overrides\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.055534 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054600 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-kubelet-config\") pod \"global-pull-secret-syncer-zzfhc\" (UID: \"9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7\") " pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:27.055534 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054616 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1da41227-eab5-43fa-a97c-69f7d935cc76-hosts-file\") pod \"node-resolver-dm6sw\" (UID: \"1da41227-eab5-43fa-a97c-69f7d935cc76\") " pod="openshift-dns/node-resolver-dm6sw" Sep 10 23:36:27.055534 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054625 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-host-var-lib-cni-bin\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.055534 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054650 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-multus-conf-dir\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.055534 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054676 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/446deb35-7fea-4d42-9fcb-c854037a9d31-device-dir\") pod \"aws-ebs-csi-driver-node-npt56\" (UID: \"446deb35-7fea-4d42-9fcb-c854037a9d31\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" Sep 10 23:36:27.055534 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054680 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-kubelet-config\") pod \"global-pull-secret-syncer-zzfhc\" (UID: \"9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7\") " pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:27.055534 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054699 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-var-lib-kubelet\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.055534 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054731 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-host-slash\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.055534 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054773 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-host-slash\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.055534 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054809 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-run-ovn\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.055534 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054842 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-ovn-node-metrics-cert\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.055534 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054846 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-var-lib-kubelet\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.055534 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054846 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-run-ovn\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.056348 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054868 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a055ce8-9e13-4aef-9d72-d14e198166a8-host-slash\") pod \"iptables-alerter-jfzt2\" (UID: \"3a055ce8-9e13-4aef-9d72-d14e198166a8\") " pod="openshift-network-operator/iptables-alerter-jfzt2" Sep 10 23:36:27.056348 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-host\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.056348 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054906 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a055ce8-9e13-4aef-9d72-d14e198166a8-host-slash\") pod \"iptables-alerter-jfzt2\" (UID: \"3a055ce8-9e13-4aef-9d72-d14e198166a8\") " pod="openshift-network-operator/iptables-alerter-jfzt2" Sep 10 23:36:27.056348 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054917 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-host-run-netns\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.056348 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054930 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1da41227-eab5-43fa-a97c-69f7d935cc76-tmp-dir\") pod \"node-resolver-dm6sw\" (UID: \"1da41227-eab5-43fa-a97c-69f7d935cc76\") " pod="openshift-dns/node-resolver-dm6sw" Sep 10 23:36:27.056348 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054958 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-ovnkube-script-lib\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.056348 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054971 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-host\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.056348 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.054982 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7be68ac6-c59b-4afa-9974-a39f8067c1f7-host\") pod \"node-ca-f7fjr\" (UID: \"7be68ac6-c59b-4afa-9974-a39f8067c1f7\") " pod="openshift-image-registry/node-ca-f7fjr" Sep 10 23:36:27.056348 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055012 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-host-run-netns\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.056348 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7be68ac6-c59b-4afa-9974-a39f8067c1f7-serviceca\") pod \"node-ca-f7fjr\" (UID: \"7be68ac6-c59b-4afa-9974-a39f8067c1f7\") " pod="openshift-image-registry/node-ca-f7fjr" Sep 10 23:36:27.056348 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055049 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4580695f-ec27-45ad-9670-4849d9354b05-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-whckt\" (UID: \"4580695f-ec27-45ad-9670-4849d9354b05\") " pod="openshift-multus/multus-additional-cni-plugins-whckt" Sep 10 23:36:27.056348 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055076 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-host-run-k8s-cni-cncf-io\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.056348 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055099 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-hostroot\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.056348 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055123 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-systemd-units\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.056348 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055121 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4b358463-cfe2-49ec-91ea-64ae0856ccd7-konnectivity-ca\") pod \"konnectivity-agent-kr8fb\" (UID: \"4b358463-cfe2-49ec-91ea-64ae0856ccd7\") " pod="kube-system/konnectivity-agent-kr8fb" Sep 10 23:36:27.056348 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055146 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-etc-openvswitch\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.056348 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055154 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-env-overrides\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.056348 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055200 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7be68ac6-c59b-4afa-9974-a39f8067c1f7-host\") pod \"node-ca-f7fjr\" (UID: \"7be68ac6-c59b-4afa-9974-a39f8067c1f7\") " pod="openshift-image-registry/node-ca-f7fjr" Sep 10 23:36:27.057181 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055171 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-etc-kubernetes\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.057181 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055224 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-systemd-units\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.057181 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055186 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 23:36:27.057181 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055261 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9v6c\" (UniqueName: \"kubernetes.io/projected/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-kube-api-access-l9v6c\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.057181 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055314 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtff6\" (UniqueName: \"kubernetes.io/projected/446deb35-7fea-4d42-9fcb-c854037a9d31-kube-api-access-vtff6\") pod \"aws-ebs-csi-driver-node-npt56\" (UID: \"446deb35-7fea-4d42-9fcb-c854037a9d31\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" Sep 10 23:36:27.057181 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055327 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-etc-openvswitch\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.057181 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055343 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4580695f-ec27-45ad-9670-4849d9354b05-cni-binary-copy\") pod \"multus-additional-cni-plugins-whckt\" (UID: \"4580695f-ec27-45ad-9670-4849d9354b05\") " pod="openshift-multus/multus-additional-cni-plugins-whckt" Sep 10 23:36:27.057181 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055377 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9drx\" (UniqueName: \"kubernetes.io/projected/c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d-kube-api-access-b9drx\") pod \"network-check-target-fb4qx\" (UID: \"c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d\") " pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:36:27.057181 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055401 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-host-run-multus-certs\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.057181 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055476 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/446deb35-7fea-4d42-9fcb-c854037a9d31-registration-dir\") pod \"aws-ebs-csi-driver-node-npt56\" (UID: \"446deb35-7fea-4d42-9fcb-c854037a9d31\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" Sep 10 23:36:27.057181 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055482 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-ovnkube-script-lib\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.057181 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055492 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7be68ac6-c59b-4afa-9974-a39f8067c1f7-serviceca\") pod \"node-ca-f7fjr\" (UID: \"7be68ac6-c59b-4afa-9974-a39f8067c1f7\") " pod="openshift-image-registry/node-ca-f7fjr" Sep 10 23:36:27.057181 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055528 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/446deb35-7fea-4d42-9fcb-c854037a9d31-kubelet-dir\") pod \"aws-ebs-csi-driver-node-npt56\" (UID: \"446deb35-7fea-4d42-9fcb-c854037a9d31\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" Sep 10 23:36:27.057181 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055556 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7czj\" (UniqueName: \"kubernetes.io/projected/8df0f6be-ce76-4263-adfb-b1531272132e-kube-api-access-b7czj\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.057181 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055582 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tkc7\" (UniqueName: \"kubernetes.io/projected/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-kube-api-access-6tkc7\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.057181 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055607 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kv9s\" (UniqueName: \"kubernetes.io/projected/1da41227-eab5-43fa-a97c-69f7d935cc76-kube-api-access-2kv9s\") pod \"node-resolver-dm6sw\" (UID: \"1da41227-eab5-43fa-a97c-69f7d935cc76\") " pod="openshift-dns/node-resolver-dm6sw" Sep 10 23:36:27.057181 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055634 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-multus-cni-dir\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.057977 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055660 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-host-var-lib-cni-multus\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.057977 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055686 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4580695f-ec27-45ad-9670-4849d9354b05-cnibin\") pod \"multus-additional-cni-plugins-whckt\" (UID: \"4580695f-ec27-45ad-9670-4849d9354b05\") " pod="openshift-multus/multus-additional-cni-plugins-whckt" Sep 10 23:36:27.057977 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055711 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-host-cni-bin\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.057977 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055735 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-ovnkube-config\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.057977 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055760 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq45v\" (UniqueName: \"kubernetes.io/projected/7be68ac6-c59b-4afa-9974-a39f8067c1f7-kube-api-access-pq45v\") pod \"node-ca-f7fjr\" (UID: \"7be68ac6-c59b-4afa-9974-a39f8067c1f7\") " pod="openshift-image-registry/node-ca-f7fjr" Sep 10 23:36:27.057977 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055786 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3a055ce8-9e13-4aef-9d72-d14e198166a8-iptables-alerter-script\") pod \"iptables-alerter-jfzt2\" (UID: \"3a055ce8-9e13-4aef-9d72-d14e198166a8\") " pod="openshift-network-operator/iptables-alerter-jfzt2" Sep 10 23:36:27.057977 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055809 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/446deb35-7fea-4d42-9fcb-c854037a9d31-etc-selinux\") pod \"aws-ebs-csi-driver-node-npt56\" (UID: \"446deb35-7fea-4d42-9fcb-c854037a9d31\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" Sep 10 23:36:27.057977 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055834 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-etc-kubernetes\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.057977 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055857 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-run-openvswitch\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.057977 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055881 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-log-socket\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.057977 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055904 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/446deb35-7fea-4d42-9fcb-c854037a9d31-socket-dir\") pod \"aws-ebs-csi-driver-node-npt56\" (UID: \"446deb35-7fea-4d42-9fcb-c854037a9d31\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" Sep 10 23:36:27.057977 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055927 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-etc-systemd\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.057977 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055973 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4b358463-cfe2-49ec-91ea-64ae0856ccd7-agent-certs\") pod \"konnectivity-agent-kr8fb\" (UID: \"4b358463-cfe2-49ec-91ea-64ae0856ccd7\") " pod="kube-system/konnectivity-agent-kr8fb" Sep 10 23:36:27.057977 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.055999 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-original-pull-secret\") pod \"global-pull-secret-syncer-zzfhc\" (UID: \"9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7\") " pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:27.057977 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.056026 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d87j6\" (UniqueName: \"kubernetes.io/projected/4580695f-ec27-45ad-9670-4849d9354b05-kube-api-access-d87j6\") pod \"multus-additional-cni-plugins-whckt\" (UID: \"4580695f-ec27-45ad-9670-4849d9354b05\") " pod="openshift-multus/multus-additional-cni-plugins-whckt" Sep 10 23:36:27.057977 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.056051 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-host-var-lib-kubelet\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.057977 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.056074 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4580695f-ec27-45ad-9670-4849d9354b05-system-cni-dir\") pod \"multus-additional-cni-plugins-whckt\" (UID: \"4580695f-ec27-45ad-9670-4849d9354b05\") " pod="openshift-multus/multus-additional-cni-plugins-whckt" Sep 10 23:36:27.058778 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.056098 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-etc-sysctl-d\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.058778 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.056125 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8df0f6be-ce76-4263-adfb-b1531272132e-etc-tuned\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.058778 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.056136 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-host-cni-bin\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.058778 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.056149 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4580695f-ec27-45ad-9670-4849d9354b05-tuning-conf-dir\") pod \"multus-additional-cni-plugins-whckt\" (UID: \"4580695f-ec27-45ad-9670-4849d9354b05\") " pod="openshift-multus/multus-additional-cni-plugins-whckt" Sep 10 23:36:27.058778 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.056173 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-cni-binary-copy\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.058778 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.056196 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-sys\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.058778 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.056218 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8df0f6be-ce76-4263-adfb-b1531272132e-tmp\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.058778 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.056241 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-host-kubelet\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.058778 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.056263 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-multus-daemon-config\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.058778 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.056285 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/446deb35-7fea-4d42-9fcb-c854037a9d31-sys-fs\") pod \"aws-ebs-csi-driver-node-npt56\" (UID: \"446deb35-7fea-4d42-9fcb-c854037a9d31\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" Sep 10 23:36:27.058778 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.056311 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-node-log\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.058778 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.056334 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-cnibin\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.058778 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.056465 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-etc-sysctl-d\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.058778 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.056528 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-ovnkube-config\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.058778 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:27.056532 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:27.058778 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:27.056599 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-original-pull-secret podName:9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:27.556583063 +0000 UTC m=+3.024057385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-original-pull-secret") pod "global-pull-secret-syncer-zzfhc" (UID: "9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7") : object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:27.058778 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.056636 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-node-log\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.059573 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.056713 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-etc-kubernetes\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.059573 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.056994 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-log-socket\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.059573 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.057012 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-run-openvswitch\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.059573 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.057070 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-sys\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.059573 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.057141 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-host-kubelet\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.059573 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.057190 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8df0f6be-ce76-4263-adfb-b1531272132e-etc-systemd\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.059573 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.057190 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3a055ce8-9e13-4aef-9d72-d14e198166a8-iptables-alerter-script\") pod \"iptables-alerter-jfzt2\" (UID: \"3a055ce8-9e13-4aef-9d72-d14e198166a8\") " pod="openshift-network-operator/iptables-alerter-jfzt2" Sep 10 23:36:27.059890 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.059684 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8df0f6be-ce76-4263-adfb-b1531272132e-tmp\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.060612 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.060382 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-ovn-node-metrics-cert\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.060612 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.060586 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8df0f6be-ce76-4263-adfb-b1531272132e-etc-tuned\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.061126 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.060974 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4b358463-cfe2-49ec-91ea-64ae0856ccd7-agent-certs\") pod \"konnectivity-agent-kr8fb\" (UID: \"4b358463-cfe2-49ec-91ea-64ae0856ccd7\") " pod="kube-system/konnectivity-agent-kr8fb" Sep 10 23:36:27.063828 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.063803 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9lpp\" (UniqueName: \"kubernetes.io/projected/a06f8008-28ac-4cb9-8310-f5c42b169d10-kube-api-access-m9lpp\") pod \"network-metrics-daemon-fmvxw\" (UID: \"a06f8008-28ac-4cb9-8310-f5c42b169d10\") " pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:27.066639 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.066617 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7lbv\" (UniqueName: \"kubernetes.io/projected/3a055ce8-9e13-4aef-9d72-d14e198166a8-kube-api-access-h7lbv\") pod \"iptables-alerter-jfzt2\" (UID: \"3a055ce8-9e13-4aef-9d72-d14e198166a8\") " pod="openshift-network-operator/iptables-alerter-jfzt2" Sep 10 23:36:27.067349 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.067266 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kv9s\" (UniqueName: \"kubernetes.io/projected/1da41227-eab5-43fa-a97c-69f7d935cc76-kube-api-access-2kv9s\") pod \"node-resolver-dm6sw\" (UID: \"1da41227-eab5-43fa-a97c-69f7d935cc76\") " pod="openshift-dns/node-resolver-dm6sw" Sep 10 23:36:27.068701 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.068680 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tkc7\" (UniqueName: \"kubernetes.io/projected/a52bd1f0-6990-4ad6-a4fd-92499e4892a2-kube-api-access-6tkc7\") pod \"ovnkube-node-8srbc\" (UID: \"a52bd1f0-6990-4ad6-a4fd-92499e4892a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.068820 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.068791 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq45v\" (UniqueName: \"kubernetes.io/projected/7be68ac6-c59b-4afa-9974-a39f8067c1f7-kube-api-access-pq45v\") pod \"node-ca-f7fjr\" (UID: \"7be68ac6-c59b-4afa-9974-a39f8067c1f7\") " pod="openshift-image-registry/node-ca-f7fjr" Sep 10 23:36:27.071330 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.071311 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7czj\" (UniqueName: \"kubernetes.io/projected/8df0f6be-ce76-4263-adfb-b1531272132e-kube-api-access-b7czj\") pod \"tuned-5pgnj\" (UID: \"8df0f6be-ce76-4263-adfb-b1531272132e\") " pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.156721 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.156686 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4580695f-ec27-45ad-9670-4849d9354b05-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-whckt\" (UID: \"4580695f-ec27-45ad-9670-4849d9354b05\") " pod="openshift-multus/multus-additional-cni-plugins-whckt" Sep 10 23:36:27.156721 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.156724 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-host-run-k8s-cni-cncf-io\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.156929 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.156745 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-hostroot\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.156929 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.156771 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-etc-kubernetes\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.156929 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.156794 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9v6c\" (UniqueName: \"kubernetes.io/projected/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-kube-api-access-l9v6c\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.156929 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.156818 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtff6\" (UniqueName: \"kubernetes.io/projected/446deb35-7fea-4d42-9fcb-c854037a9d31-kube-api-access-vtff6\") pod \"aws-ebs-csi-driver-node-npt56\" (UID: \"446deb35-7fea-4d42-9fcb-c854037a9d31\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" Sep 10 23:36:27.156929 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.156822 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-host-run-k8s-cni-cncf-io\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.156929 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.156834 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4580695f-ec27-45ad-9670-4849d9354b05-cni-binary-copy\") pod \"multus-additional-cni-plugins-whckt\" (UID: \"4580695f-ec27-45ad-9670-4849d9354b05\") " pod="openshift-multus/multus-additional-cni-plugins-whckt" Sep 10 23:36:27.156929 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.156883 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9drx\" (UniqueName: \"kubernetes.io/projected/c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d-kube-api-access-b9drx\") pod \"network-check-target-fb4qx\" (UID: \"c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d\") " pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:36:27.156929 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.156888 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-hostroot\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.156929 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.156910 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-host-run-multus-certs\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.156929 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.156949 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/446deb35-7fea-4d42-9fcb-c854037a9d31-registration-dir\") pod \"aws-ebs-csi-driver-node-npt56\" (UID: \"446deb35-7fea-4d42-9fcb-c854037a9d31\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" Sep 10 23:36:27.157398 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.156976 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/446deb35-7fea-4d42-9fcb-c854037a9d31-kubelet-dir\") pod \"aws-ebs-csi-driver-node-npt56\" (UID: \"446deb35-7fea-4d42-9fcb-c854037a9d31\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" Sep 10 23:36:27.157398 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157005 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-multus-cni-dir\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.157398 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157030 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-host-var-lib-cni-multus\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.157398 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.156951 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-etc-kubernetes\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.157398 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157072 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-host-var-lib-cni-multus\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.157398 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157088 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-host-run-multus-certs\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.157398 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157102 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/446deb35-7fea-4d42-9fcb-c854037a9d31-registration-dir\") pod \"aws-ebs-csi-driver-node-npt56\" (UID: \"446deb35-7fea-4d42-9fcb-c854037a9d31\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" Sep 10 23:36:27.157398 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157138 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/446deb35-7fea-4d42-9fcb-c854037a9d31-kubelet-dir\") pod \"aws-ebs-csi-driver-node-npt56\" (UID: \"446deb35-7fea-4d42-9fcb-c854037a9d31\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" Sep 10 23:36:27.157398 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157168 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4580695f-ec27-45ad-9670-4849d9354b05-cnibin\") pod \"multus-additional-cni-plugins-whckt\" (UID: \"4580695f-ec27-45ad-9670-4849d9354b05\") " pod="openshift-multus/multus-additional-cni-plugins-whckt" Sep 10 23:36:27.157398 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157202 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/446deb35-7fea-4d42-9fcb-c854037a9d31-etc-selinux\") pod \"aws-ebs-csi-driver-node-npt56\" (UID: \"446deb35-7fea-4d42-9fcb-c854037a9d31\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" Sep 10 23:36:27.157398 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157206 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4580695f-ec27-45ad-9670-4849d9354b05-cnibin\") pod \"multus-additional-cni-plugins-whckt\" (UID: \"4580695f-ec27-45ad-9670-4849d9354b05\") " pod="openshift-multus/multus-additional-cni-plugins-whckt" Sep 10 23:36:27.157398 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157169 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-multus-cni-dir\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.157398 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157230 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/446deb35-7fea-4d42-9fcb-c854037a9d31-socket-dir\") pod \"aws-ebs-csi-driver-node-npt56\" (UID: \"446deb35-7fea-4d42-9fcb-c854037a9d31\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" Sep 10 23:36:27.157398 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157312 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d87j6\" (UniqueName: \"kubernetes.io/projected/4580695f-ec27-45ad-9670-4849d9354b05-kube-api-access-d87j6\") pod \"multus-additional-cni-plugins-whckt\" (UID: \"4580695f-ec27-45ad-9670-4849d9354b05\") " pod="openshift-multus/multus-additional-cni-plugins-whckt" Sep 10 23:36:27.157398 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157333 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/446deb35-7fea-4d42-9fcb-c854037a9d31-etc-selinux\") pod \"aws-ebs-csi-driver-node-npt56\" (UID: \"446deb35-7fea-4d42-9fcb-c854037a9d31\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" Sep 10 23:36:27.157398 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157351 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-host-var-lib-kubelet\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.157398 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157349 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4580695f-ec27-45ad-9670-4849d9354b05-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-whckt\" (UID: \"4580695f-ec27-45ad-9670-4849d9354b05\") " pod="openshift-multus/multus-additional-cni-plugins-whckt" Sep 10 23:36:27.158168 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157368 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/446deb35-7fea-4d42-9fcb-c854037a9d31-socket-dir\") pod \"aws-ebs-csi-driver-node-npt56\" (UID: \"446deb35-7fea-4d42-9fcb-c854037a9d31\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" Sep 10 23:36:27.158168 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157376 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4580695f-ec27-45ad-9670-4849d9354b05-system-cni-dir\") pod \"multus-additional-cni-plugins-whckt\" (UID: \"4580695f-ec27-45ad-9670-4849d9354b05\") " pod="openshift-multus/multus-additional-cni-plugins-whckt" Sep 10 23:36:27.158168 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157412 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4580695f-ec27-45ad-9670-4849d9354b05-system-cni-dir\") pod \"multus-additional-cni-plugins-whckt\" (UID: \"4580695f-ec27-45ad-9670-4849d9354b05\") " pod="openshift-multus/multus-additional-cni-plugins-whckt" Sep 10 23:36:27.158168 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157417 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-host-var-lib-kubelet\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.158168 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157448 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4580695f-ec27-45ad-9670-4849d9354b05-tuning-conf-dir\") pod \"multus-additional-cni-plugins-whckt\" (UID: \"4580695f-ec27-45ad-9670-4849d9354b05\") " pod="openshift-multus/multus-additional-cni-plugins-whckt" Sep 10 23:36:27.158168 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157529 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4580695f-ec27-45ad-9670-4849d9354b05-cni-binary-copy\") pod \"multus-additional-cni-plugins-whckt\" (UID: \"4580695f-ec27-45ad-9670-4849d9354b05\") " pod="openshift-multus/multus-additional-cni-plugins-whckt" Sep 10 23:36:27.158168 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157543 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-cni-binary-copy\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.158168 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157579 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-multus-daemon-config\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.158168 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157585 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4580695f-ec27-45ad-9670-4849d9354b05-tuning-conf-dir\") pod \"multus-additional-cni-plugins-whckt\" (UID: \"4580695f-ec27-45ad-9670-4849d9354b05\") " pod="openshift-multus/multus-additional-cni-plugins-whckt" Sep 10 23:36:27.158168 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157605 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/446deb35-7fea-4d42-9fcb-c854037a9d31-sys-fs\") pod \"aws-ebs-csi-driver-node-npt56\" (UID: \"446deb35-7fea-4d42-9fcb-c854037a9d31\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" Sep 10 23:36:27.158168 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157630 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-cnibin\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.158168 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157654 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-multus-socket-dir-parent\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.158168 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157696 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-system-cni-dir\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.158168 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157706 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/446deb35-7fea-4d42-9fcb-c854037a9d31-sys-fs\") pod \"aws-ebs-csi-driver-node-npt56\" (UID: \"446deb35-7fea-4d42-9fcb-c854037a9d31\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" Sep 10 23:36:27.158168 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157718 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-cnibin\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.158168 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157724 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4580695f-ec27-45ad-9670-4849d9354b05-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-whckt\" (UID: \"4580695f-ec27-45ad-9670-4849d9354b05\") " pod="openshift-multus/multus-additional-cni-plugins-whckt" Sep 10 23:36:27.158168 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157764 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-multus-socket-dir-parent\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.158722 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157771 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-system-cni-dir\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.158722 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157781 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-host-run-netns\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.158722 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157838 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-host-var-lib-cni-bin\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.158722 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157866 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-multus-conf-dir\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.158722 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157878 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-host-run-netns\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.158722 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157889 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/446deb35-7fea-4d42-9fcb-c854037a9d31-device-dir\") pod \"aws-ebs-csi-driver-node-npt56\" (UID: \"446deb35-7fea-4d42-9fcb-c854037a9d31\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" Sep 10 23:36:27.158722 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157926 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-host-var-lib-cni-bin\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.158722 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157952 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-multus-conf-dir\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.158722 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.157985 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/446deb35-7fea-4d42-9fcb-c854037a9d31-device-dir\") pod \"aws-ebs-csi-driver-node-npt56\" (UID: \"446deb35-7fea-4d42-9fcb-c854037a9d31\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" Sep 10 23:36:27.158722 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.158011 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-cni-binary-copy\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.158722 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.158092 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-multus-daemon-config\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.158722 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.158203 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4580695f-ec27-45ad-9670-4849d9354b05-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-whckt\" (UID: \"4580695f-ec27-45ad-9670-4849d9354b05\") " pod="openshift-multus/multus-additional-cni-plugins-whckt" Sep 10 23:36:27.164007 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:27.163875 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 23:36:27.164007 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:27.163899 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 23:36:27.164007 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:27.163912 2578 projected.go:196] Error preparing data for projected volume kube-api-access-b9drx for pod openshift-network-diagnostics/network-check-target-fb4qx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:27.164224 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:27.164010 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d-kube-api-access-b9drx podName:c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d nodeName:}" failed. No retries permitted until 2026-09-10 23:36:27.663991327 +0000 UTC m=+3.131465648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-b9drx" (UniqueName: "kubernetes.io/projected/c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d-kube-api-access-b9drx") pod "network-check-target-fb4qx" (UID: "c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:27.165895 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.165876 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtff6\" (UniqueName: \"kubernetes.io/projected/446deb35-7fea-4d42-9fcb-c854037a9d31-kube-api-access-vtff6\") pod \"aws-ebs-csi-driver-node-npt56\" (UID: \"446deb35-7fea-4d42-9fcb-c854037a9d31\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" Sep 10 23:36:27.166077 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.165971 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d87j6\" (UniqueName: \"kubernetes.io/projected/4580695f-ec27-45ad-9670-4849d9354b05-kube-api-access-d87j6\") pod \"multus-additional-cni-plugins-whckt\" (UID: \"4580695f-ec27-45ad-9670-4849d9354b05\") " pod="openshift-multus/multus-additional-cni-plugins-whckt" Sep 10 23:36:27.166280 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.166264 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9v6c\" (UniqueName: \"kubernetes.io/projected/4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd-kube-api-access-l9v6c\") pod \"multus-v77rf\" (UID: \"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd\") " pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.241822 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.241775 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jfzt2" Sep 10 23:36:27.255137 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.255106 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" Sep 10 23:36:27.266035 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.266009 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:27.271641 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.271622 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kr8fb" Sep 10 23:36:27.280393 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.280367 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dm6sw" Sep 10 23:36:27.290079 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.290056 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f7fjr" Sep 10 23:36:27.296629 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.296610 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-whckt" Sep 10 23:36:27.302992 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.302973 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 23:36:27.304823 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.304805 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v77rf" Sep 10 23:36:27.312498 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.312483 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" Sep 10 23:36:27.560619 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.560540 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-original-pull-secret\") pod \"global-pull-secret-syncer-zzfhc\" (UID: \"9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7\") " pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:27.560619 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.560582 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a06f8008-28ac-4cb9-8310-f5c42b169d10-metrics-certs\") pod \"network-metrics-daemon-fmvxw\" (UID: \"a06f8008-28ac-4cb9-8310-f5c42b169d10\") " pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:27.560821 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:27.560679 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:27.560821 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:27.560682 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:27.560821 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:27.560749 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a06f8008-28ac-4cb9-8310-f5c42b169d10-metrics-certs podName:a06f8008-28ac-4cb9-8310-f5c42b169d10 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:28.560731061 +0000 UTC m=+4.028205387 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a06f8008-28ac-4cb9-8310-f5c42b169d10-metrics-certs") pod "network-metrics-daemon-fmvxw" (UID: "a06f8008-28ac-4cb9-8310-f5c42b169d10") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:27.560821 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:27.560767 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-original-pull-secret podName:9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:28.560758697 +0000 UTC m=+4.028233004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-original-pull-secret") pod "global-pull-secret-syncer-zzfhc" (UID: "9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7") : object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:27.762003 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.761966 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9drx\" (UniqueName: \"kubernetes.io/projected/c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d-kube-api-access-b9drx\") pod \"network-check-target-fb4qx\" (UID: \"c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d\") " pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:36:27.762179 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:27.762155 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 23:36:27.762227 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:27.762190 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 23:36:27.762227 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:27.762204 2578 projected.go:196] Error preparing data for projected volume kube-api-access-b9drx for pod openshift-network-diagnostics/network-check-target-fb4qx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:27.762294 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:27.762275 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d-kube-api-access-b9drx podName:c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d nodeName:}" failed. No retries permitted until 2026-09-10 23:36:28.762255769 +0000 UTC m=+4.229730075 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-b9drx" (UniqueName: "kubernetes.io/projected/c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d-kube-api-access-b9drx") pod "network-check-target-fb4qx" (UID: "c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:27.983073 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.982989 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 23:31:26 +0000 UTC" deadline="2028-05-25 17:40:05.561742814 +0000 UTC" Sep 10 23:36:27.983073 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:27.983021 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14946h3m37.578726295s" Sep 10 23:36:28.038498 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:28.038459 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:28.038671 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:28.038585 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmvxw" podUID="a06f8008-28ac-4cb9-8310-f5c42b169d10" Sep 10 23:36:28.092189 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:28.092163 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7be68ac6_c59b_4afa_9974_a39f8067c1f7.slice/crio-29a8b77aa429b09db6601c287819d734a4c1cd7fb4b2784b94e3d2b7720fab6d WatchSource:0}: Error finding container 29a8b77aa429b09db6601c287819d734a4c1cd7fb4b2784b94e3d2b7720fab6d: Status 404 returned error can't find the container with id 29a8b77aa429b09db6601c287819d734a4c1cd7fb4b2784b94e3d2b7720fab6d Sep 10 23:36:28.097353 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:28.097327 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df0f6be_ce76_4263_adfb_b1531272132e.slice/crio-86f540364a7065ac642acfeae51c2d1276fd0cba6f5f1e3e248341e04be776c4 WatchSource:0}: Error finding container 86f540364a7065ac642acfeae51c2d1276fd0cba6f5f1e3e248341e04be776c4: Status 404 returned error can't find the container with id 86f540364a7065ac642acfeae51c2d1276fd0cba6f5f1e3e248341e04be776c4 Sep 10 23:36:28.098182 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:28.098159 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b358463_cfe2_49ec_91ea_64ae0856ccd7.slice/crio-7114dfb4617efc7184d0f03be2872f337585917179c8027c6b4eb1c21af3b21f WatchSource:0}: Error finding container 7114dfb4617efc7184d0f03be2872f337585917179c8027c6b4eb1c21af3b21f: Status 404 returned error can't find the container with id 7114dfb4617efc7184d0f03be2872f337585917179c8027c6b4eb1c21af3b21f Sep 10 23:36:28.100132 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:28.099974 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da41227_eab5_43fa_a97c_69f7d935cc76.slice/crio-f3d96c87d37f8a92297d253ba766f1564008b9a9e3f17b8d779eaca83ebc06a0 WatchSource:0}: Error finding container f3d96c87d37f8a92297d253ba766f1564008b9a9e3f17b8d779eaca83ebc06a0: Status 404 returned error can't find the container with id f3d96c87d37f8a92297d253ba766f1564008b9a9e3f17b8d779eaca83ebc06a0 Sep 10 23:36:28.100671 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:28.100640 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda52bd1f0_6990_4ad6_a4fd_92499e4892a2.slice/crio-a2663648f6bac3d5272eeecfdb837d4ec57653dbd6640523e01bcf2d99727b59 WatchSource:0}: Error finding container a2663648f6bac3d5272eeecfdb837d4ec57653dbd6640523e01bcf2d99727b59: Status 404 returned error can't find the container with id a2663648f6bac3d5272eeecfdb837d4ec57653dbd6640523e01bcf2d99727b59 Sep 10 23:36:28.101589 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:28.101456 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e1b3ec6_1e96_4306_ac02_5aa1a56d37bd.slice/crio-c03cf1bb84bfb1e3f446286b3f3e32a26983aaa6b7eb74c938d1fa41c5caab96 WatchSource:0}: Error finding container c03cf1bb84bfb1e3f446286b3f3e32a26983aaa6b7eb74c938d1fa41c5caab96: Status 404 returned error can't find the container with id c03cf1bb84bfb1e3f446286b3f3e32a26983aaa6b7eb74c938d1fa41c5caab96 Sep 10 23:36:28.102955 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:36:28.102909 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod446deb35_7fea_4d42_9fcb_c854037a9d31.slice/crio-918365a1f53d27cf047e8188cacb3ba7156a78d114ff1bad50da1897623883b6 WatchSource:0}: Error finding container 918365a1f53d27cf047e8188cacb3ba7156a78d114ff1bad50da1897623883b6: Status 404 returned error can't find the container with id 918365a1f53d27cf047e8188cacb3ba7156a78d114ff1bad50da1897623883b6 Sep 10 23:36:28.567602 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:28.566606 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-original-pull-secret\") pod \"global-pull-secret-syncer-zzfhc\" (UID: \"9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7\") " pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:28.567602 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:28.566881 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a06f8008-28ac-4cb9-8310-f5c42b169d10-metrics-certs\") pod \"network-metrics-daemon-fmvxw\" (UID: \"a06f8008-28ac-4cb9-8310-f5c42b169d10\") " pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:28.567602 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:28.567063 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:28.567602 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:28.567127 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a06f8008-28ac-4cb9-8310-f5c42b169d10-metrics-certs podName:a06f8008-28ac-4cb9-8310-f5c42b169d10 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:30.567108666 +0000 UTC m=+6.034582989 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a06f8008-28ac-4cb9-8310-f5c42b169d10-metrics-certs") pod "network-metrics-daemon-fmvxw" (UID: "a06f8008-28ac-4cb9-8310-f5c42b169d10") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:28.567602 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:28.567504 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:28.567602 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:28.567563 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-original-pull-secret podName:9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:30.567538787 +0000 UTC m=+6.035013093 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-original-pull-secret") pod "global-pull-secret-syncer-zzfhc" (UID: "9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7") : object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:28.768754 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:28.768637 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9drx\" (UniqueName: \"kubernetes.io/projected/c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d-kube-api-access-b9drx\") pod \"network-check-target-fb4qx\" (UID: \"c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d\") " pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:36:28.768910 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:28.768831 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 23:36:28.768910 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:28.768852 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 23:36:28.768910 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:28.768864 2578 projected.go:196] Error preparing data for projected volume kube-api-access-b9drx for pod openshift-network-diagnostics/network-check-target-fb4qx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:28.769103 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:28.768924 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d-kube-api-access-b9drx podName:c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d nodeName:}" failed. No retries permitted until 2026-09-10 23:36:30.768905797 +0000 UTC m=+6.236380103 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-b9drx" (UniqueName: "kubernetes.io/projected/c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d-kube-api-access-b9drx") pod "network-check-target-fb4qx" (UID: "c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:28.984220 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:28.984172 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 23:31:26 +0000 UTC" deadline="2028-02-09 13:58:05.819916839 +0000 UTC" Sep 10 23:36:28.984220 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:28.984216 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12398h21m36.835705138s" Sep 10 23:36:29.039602 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:29.039054 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:29.039602 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:29.039177 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzfhc" podUID="9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7" Sep 10 23:36:29.040153 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:29.039922 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:36:29.040153 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:29.040103 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fb4qx" podUID="c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d" Sep 10 23:36:29.058822 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:29.058737 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kr8fb" event={"ID":"4b358463-cfe2-49ec-91ea-64ae0856ccd7","Type":"ContainerStarted","Data":"7114dfb4617efc7184d0f03be2872f337585917179c8027c6b4eb1c21af3b21f"} Sep 10 23:36:29.062282 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:29.062197 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" event={"ID":"8df0f6be-ce76-4263-adfb-b1531272132e","Type":"ContainerStarted","Data":"86f540364a7065ac642acfeae51c2d1276fd0cba6f5f1e3e248341e04be776c4"} Sep 10 23:36:29.066355 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:29.066175 2578 generic.go:356] "Generic (PLEG): container finished" podID="6965b1dacc260bd4d7bb7a14199bc848" containerID="8fd1ea68c6350b902ec063972707e774112e0782099e5762cd9ecc3b75f2ed3a" exitCode=0 Sep 10 23:36:29.066355 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:29.066250 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal" event={"ID":"6965b1dacc260bd4d7bb7a14199bc848","Type":"ContainerDied","Data":"8fd1ea68c6350b902ec063972707e774112e0782099e5762cd9ecc3b75f2ed3a"} Sep 10 23:36:29.079665 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:29.079540 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v77rf" event={"ID":"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd","Type":"ContainerStarted","Data":"c03cf1bb84bfb1e3f446286b3f3e32a26983aaa6b7eb74c938d1fa41c5caab96"} Sep 10 23:36:29.091986 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:29.091899 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f7fjr" event={"ID":"7be68ac6-c59b-4afa-9974-a39f8067c1f7","Type":"ContainerStarted","Data":"29a8b77aa429b09db6601c287819d734a4c1cd7fb4b2784b94e3d2b7720fab6d"} Sep 10 23:36:29.098524 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:29.098495 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-142.ec2.internal" event={"ID":"bff3cfe4b5e3fbd1418d844a741d3b63","Type":"ContainerStarted","Data":"3411501b42ab96a1ef68cdbeb3e119157fd8aaceada6ee9345f18be864257593"} Sep 10 23:36:29.107420 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:29.107368 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jfzt2" event={"ID":"3a055ce8-9e13-4aef-9d72-d14e198166a8","Type":"ContainerStarted","Data":"b2ad410ae0b39b3843c2ac1386b445cb39cd0f49b657fa594ae912f958d59902"} Sep 10 23:36:29.112875 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:29.112035 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-142.ec2.internal" podStartSLOduration=3.112016752 podStartE2EDuration="3.112016752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:29.111610379 +0000 UTC m=+4.579084707" watchObservedRunningTime="2026-09-10 23:36:29.112016752 +0000 UTC m=+4.579491079" Sep 10 23:36:29.122771 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:29.122642 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" event={"ID":"446deb35-7fea-4d42-9fcb-c854037a9d31","Type":"ContainerStarted","Data":"918365a1f53d27cf047e8188cacb3ba7156a78d114ff1bad50da1897623883b6"} Sep 10 23:36:29.138560 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:29.138429 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-whckt" event={"ID":"4580695f-ec27-45ad-9670-4849d9354b05","Type":"ContainerStarted","Data":"b705a7b20c354189692a135b8dd2cf5a3f098d8ea205a537a5d3ce914b6db93b"} Sep 10 23:36:29.140977 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:29.140852 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" event={"ID":"a52bd1f0-6990-4ad6-a4fd-92499e4892a2","Type":"ContainerStarted","Data":"a2663648f6bac3d5272eeecfdb837d4ec57653dbd6640523e01bcf2d99727b59"} Sep 10 23:36:29.146349 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:29.146323 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dm6sw" event={"ID":"1da41227-eab5-43fa-a97c-69f7d935cc76","Type":"ContainerStarted","Data":"f3d96c87d37f8a92297d253ba766f1564008b9a9e3f17b8d779eaca83ebc06a0"} Sep 10 23:36:30.038784 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:30.038700 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:30.039324 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:30.038850 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmvxw" podUID="a06f8008-28ac-4cb9-8310-f5c42b169d10" Sep 10 23:36:30.169032 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:30.169001 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal" event={"ID":"6965b1dacc260bd4d7bb7a14199bc848","Type":"ContainerStarted","Data":"5f7ae70489fcf8d42a2521d98ba8d586bcd8d528c5c97297dc1720d873e540db"} Sep 10 23:36:30.587448 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:30.586739 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-original-pull-secret\") pod \"global-pull-secret-syncer-zzfhc\" (UID: \"9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7\") " pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:30.587448 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:30.586801 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a06f8008-28ac-4cb9-8310-f5c42b169d10-metrics-certs\") pod \"network-metrics-daemon-fmvxw\" (UID: \"a06f8008-28ac-4cb9-8310-f5c42b169d10\") " pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:30.587448 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:30.586955 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:30.587448 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:30.587022 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a06f8008-28ac-4cb9-8310-f5c42b169d10-metrics-certs podName:a06f8008-28ac-4cb9-8310-f5c42b169d10 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:34.587002125 +0000 UTC m=+10.054476432 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a06f8008-28ac-4cb9-8310-f5c42b169d10-metrics-certs") pod "network-metrics-daemon-fmvxw" (UID: "a06f8008-28ac-4cb9-8310-f5c42b169d10") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:30.587448 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:30.587088 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:30.587448 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:30.587123 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-original-pull-secret podName:9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:34.587111532 +0000 UTC m=+10.054585835 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-original-pull-secret") pod "global-pull-secret-syncer-zzfhc" (UID: "9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7") : object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:30.789631 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:30.788969 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9drx\" (UniqueName: \"kubernetes.io/projected/c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d-kube-api-access-b9drx\") pod \"network-check-target-fb4qx\" (UID: \"c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d\") " pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:36:30.789631 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:30.789190 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 23:36:30.789631 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:30.789212 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 23:36:30.789631 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:30.789226 2578 projected.go:196] Error preparing data for projected volume kube-api-access-b9drx for pod openshift-network-diagnostics/network-check-target-fb4qx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:30.789631 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:30.789283 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d-kube-api-access-b9drx podName:c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d nodeName:}" failed. No retries permitted until 2026-09-10 23:36:34.789265134 +0000 UTC m=+10.256739461 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-b9drx" (UniqueName: "kubernetes.io/projected/c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d-kube-api-access-b9drx") pod "network-check-target-fb4qx" (UID: "c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:31.039147 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:31.039057 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:31.039563 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:31.039204 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzfhc" podUID="9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7" Sep 10 23:36:31.039563 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:31.039250 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:36:31.039563 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:31.039366 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fb4qx" podUID="c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d" Sep 10 23:36:32.038794 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:32.038759 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:32.039008 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:32.038908 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmvxw" podUID="a06f8008-28ac-4cb9-8310-f5c42b169d10" Sep 10 23:36:33.039161 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:33.039125 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:36:33.039618 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:33.039299 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fb4qx" podUID="c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d" Sep 10 23:36:33.039618 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:33.039365 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:33.039618 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:33.039435 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzfhc" podUID="9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7" Sep 10 23:36:34.039374 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:34.038802 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:34.039374 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:34.038960 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmvxw" podUID="a06f8008-28ac-4cb9-8310-f5c42b169d10" Sep 10 23:36:34.622487 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:34.622452 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-original-pull-secret\") pod \"global-pull-secret-syncer-zzfhc\" (UID: \"9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7\") " pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:34.622674 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:34.622514 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a06f8008-28ac-4cb9-8310-f5c42b169d10-metrics-certs\") pod \"network-metrics-daemon-fmvxw\" (UID: \"a06f8008-28ac-4cb9-8310-f5c42b169d10\") " pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:34.622674 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:34.622659 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:34.622787 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:34.622714 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a06f8008-28ac-4cb9-8310-f5c42b169d10-metrics-certs podName:a06f8008-28ac-4cb9-8310-f5c42b169d10 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:42.622695613 +0000 UTC m=+18.090169931 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a06f8008-28ac-4cb9-8310-f5c42b169d10-metrics-certs") pod "network-metrics-daemon-fmvxw" (UID: "a06f8008-28ac-4cb9-8310-f5c42b169d10") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:34.623104 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:34.623015 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:34.623104 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:34.623084 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-original-pull-secret podName:9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:42.623065418 +0000 UTC m=+18.090539739 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-original-pull-secret") pod "global-pull-secret-syncer-zzfhc" (UID: "9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7") : object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:34.824362 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:34.824282 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9drx\" (UniqueName: \"kubernetes.io/projected/c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d-kube-api-access-b9drx\") pod \"network-check-target-fb4qx\" (UID: \"c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d\") " pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:36:34.824536 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:34.824398 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 23:36:34.824536 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:34.824417 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 23:36:34.824536 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:34.824429 2578 projected.go:196] Error preparing data for projected volume kube-api-access-b9drx for pod openshift-network-diagnostics/network-check-target-fb4qx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:34.824536 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:34.824492 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d-kube-api-access-b9drx podName:c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d nodeName:}" failed. No retries permitted until 2026-09-10 23:36:42.824475973 +0000 UTC m=+18.291950293 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-b9drx" (UniqueName: "kubernetes.io/projected/c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d-kube-api-access-b9drx") pod "network-check-target-fb4qx" (UID: "c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:35.039881 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:35.039409 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:35.039881 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:35.039515 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzfhc" podUID="9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7" Sep 10 23:36:35.039881 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:35.039570 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:36:35.039881 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:35.039653 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fb4qx" podUID="c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d" Sep 10 23:36:36.038835 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:36.038802 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:36.039030 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:36.038956 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmvxw" podUID="a06f8008-28ac-4cb9-8310-f5c42b169d10" Sep 10 23:36:37.038528 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:37.038490 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:36:37.038968 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:37.038497 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:37.038968 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:37.038618 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fb4qx" podUID="c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d" Sep 10 23:36:37.038968 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:37.038697 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzfhc" podUID="9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7" Sep 10 23:36:38.038981 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:38.038914 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:38.039413 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:38.039064 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmvxw" podUID="a06f8008-28ac-4cb9-8310-f5c42b169d10" Sep 10 23:36:39.039068 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:39.038975 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:39.039523 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:39.039106 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzfhc" podUID="9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7" Sep 10 23:36:39.039523 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:39.039162 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:36:39.039523 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:39.039272 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fb4qx" podUID="c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d" Sep 10 23:36:40.038836 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:40.038799 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:40.039050 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:40.038960 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmvxw" podUID="a06f8008-28ac-4cb9-8310-f5c42b169d10" Sep 10 23:36:41.038631 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:41.038601 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:36:41.039053 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:41.038761 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fb4qx" podUID="c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d" Sep 10 23:36:41.039053 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:41.038806 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:41.039053 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:41.038905 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzfhc" podUID="9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7" Sep 10 23:36:42.038544 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:42.038512 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:42.038723 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:42.038629 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmvxw" podUID="a06f8008-28ac-4cb9-8310-f5c42b169d10" Sep 10 23:36:42.676137 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:42.676042 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-original-pull-secret\") pod \"global-pull-secret-syncer-zzfhc\" (UID: \"9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7\") " pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:42.676137 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:42.676112 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a06f8008-28ac-4cb9-8310-f5c42b169d10-metrics-certs\") pod \"network-metrics-daemon-fmvxw\" (UID: \"a06f8008-28ac-4cb9-8310-f5c42b169d10\") " pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:42.676331 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:42.676252 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:42.676331 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:42.676322 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a06f8008-28ac-4cb9-8310-f5c42b169d10-metrics-certs podName:a06f8008-28ac-4cb9-8310-f5c42b169d10 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:58.676301895 +0000 UTC m=+34.143776216 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a06f8008-28ac-4cb9-8310-f5c42b169d10-metrics-certs") pod "network-metrics-daemon-fmvxw" (UID: "a06f8008-28ac-4cb9-8310-f5c42b169d10") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:42.676823 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:42.676799 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:42.676966 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:42.676862 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-original-pull-secret podName:9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:58.676845914 +0000 UTC m=+34.144320231 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-original-pull-secret") pod "global-pull-secret-syncer-zzfhc" (UID: "9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7") : object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:42.878199 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:42.878167 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9drx\" (UniqueName: \"kubernetes.io/projected/c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d-kube-api-access-b9drx\") pod \"network-check-target-fb4qx\" (UID: \"c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d\") " pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:36:42.878381 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:42.878305 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 23:36:42.878381 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:42.878320 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 23:36:42.878381 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:42.878328 2578 projected.go:196] Error preparing data for projected volume kube-api-access-b9drx for pod openshift-network-diagnostics/network-check-target-fb4qx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:42.878508 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:42.878389 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d-kube-api-access-b9drx podName:c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d nodeName:}" failed. No retries permitted until 2026-09-10 23:36:58.87837363 +0000 UTC m=+34.345847933 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-b9drx" (UniqueName: "kubernetes.io/projected/c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d-kube-api-access-b9drx") pod "network-check-target-fb4qx" (UID: "c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:43.038850 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:43.038763 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:43.039299 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:43.038897 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzfhc" podUID="9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7" Sep 10 23:36:43.039299 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:43.038954 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:36:43.039299 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:43.039066 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fb4qx" podUID="c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d" Sep 10 23:36:44.039009 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:44.038968 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:44.039502 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:44.039114 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmvxw" podUID="a06f8008-28ac-4cb9-8310-f5c42b169d10" Sep 10 23:36:45.041770 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:45.041742 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:45.042234 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:45.041745 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:36:45.042234 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:45.041862 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzfhc" podUID="9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7" Sep 10 23:36:45.042234 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:45.041928 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fb4qx" podUID="c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d" Sep 10 23:36:46.038018 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:46.037989 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:46.038152 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:46.038126 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmvxw" podUID="a06f8008-28ac-4cb9-8310-f5c42b169d10" Sep 10 23:36:46.201171 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:46.201103 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" event={"ID":"446deb35-7fea-4d42-9fcb-c854037a9d31","Type":"ContainerStarted","Data":"21c2ca0d94102806d3a593334fae85837449fc0218cec109e10b5a8ebc7cc0e9"} Sep 10 23:36:46.202560 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:46.202529 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-whckt" event={"ID":"4580695f-ec27-45ad-9670-4849d9354b05","Type":"ContainerStarted","Data":"0157b83ed29e9bd7d6a98ea96041e151d1dbeb10f9ea16693912d6c8c93b45a2"} Sep 10 23:36:46.203894 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:46.203875 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kr8fb" event={"ID":"4b358463-cfe2-49ec-91ea-64ae0856ccd7","Type":"ContainerStarted","Data":"6949da5830d1b92f6a624c1e34f522e78e8fb93e78030293fe4c18659f41a870"} Sep 10 23:36:46.205141 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:46.205122 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f7fjr" event={"ID":"7be68ac6-c59b-4afa-9974-a39f8067c1f7","Type":"ContainerStarted","Data":"4be7387e51484dd33d99eec94e5a66c9d5a5c487c4a6c5974a64e13bc02a5615"} Sep 10 23:36:46.223490 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:46.223452 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-142.ec2.internal" podStartSLOduration=20.223441063 podStartE2EDuration="20.223441063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:30.184159961 +0000 UTC m=+5.651634287" watchObservedRunningTime="2026-09-10 23:36:46.223441063 +0000 UTC m=+21.690915390" Sep 10 23:36:46.235108 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:46.235067 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kr8fb" podStartSLOduration=7.059421739 podStartE2EDuration="21.235052961s" totalImagesPullingTime="14.175631222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:46.23458658 +0000 UTC m=+21.702060906" watchObservedRunningTime="2026-09-10 23:36:46.235052961 +0000 UTC m=+21.702527287" Sep 10 23:36:47.043983 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:47.043783 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:36:47.044174 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:47.043850 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:47.044174 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:47.044138 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzfhc" podUID="9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7" Sep 10 23:36:47.044174 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:47.044037 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fb4qx" podUID="c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d" Sep 10 23:36:47.208772 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:47.208732 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v77rf" event={"ID":"4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd","Type":"ContainerStarted","Data":"6e4b95cf46c06c1f466309d8d6abb164fb38d4d779501386607126f9341c0d2e"} Sep 10 23:36:47.211539 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:47.211511 2578 generic.go:356] "Generic (PLEG): container finished" podID="4580695f-ec27-45ad-9670-4849d9354b05" containerID="0157b83ed29e9bd7d6a98ea96041e151d1dbeb10f9ea16693912d6c8c93b45a2" exitCode=0 Sep 10 23:36:47.211685 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:47.211601 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-whckt" event={"ID":"4580695f-ec27-45ad-9670-4849d9354b05","Type":"ContainerDied","Data":"0157b83ed29e9bd7d6a98ea96041e151d1dbeb10f9ea16693912d6c8c93b45a2"} Sep 10 23:36:47.214788 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:47.214761 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" event={"ID":"a52bd1f0-6990-4ad6-a4fd-92499e4892a2","Type":"ContainerStarted","Data":"c43e4bb4c614992d851211475c4c1fe9d5d445b8c7581b26f7f87e1a9e96ba98"} Sep 10 23:36:47.214896 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:47.214827 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" event={"ID":"a52bd1f0-6990-4ad6-a4fd-92499e4892a2","Type":"ContainerStarted","Data":"b24e1400bc184a0b1713ca9186f0a8a1761bba225555b3bb870103c890d0cb72"} Sep 10 23:36:47.214896 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:47.214858 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" event={"ID":"a52bd1f0-6990-4ad6-a4fd-92499e4892a2","Type":"ContainerStarted","Data":"29cc31622e07040a9792cc2edee5d5d15d0470d479a8bf7fd56557f898c2b657"} Sep 10 23:36:47.215006 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:47.214897 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" event={"ID":"a52bd1f0-6990-4ad6-a4fd-92499e4892a2","Type":"ContainerStarted","Data":"1a27f68d11ee67bf2ca34551132f7ffb822b3700707731ef53b4d2a106c34e7d"} Sep 10 23:36:47.215006 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:47.214949 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" event={"ID":"a52bd1f0-6990-4ad6-a4fd-92499e4892a2","Type":"ContainerStarted","Data":"f19a2109a238e1d7797b4c432420682b5c131b70d4f11b099d50291edcc27edb"} Sep 10 23:36:47.215006 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:47.214983 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" event={"ID":"a52bd1f0-6990-4ad6-a4fd-92499e4892a2","Type":"ContainerStarted","Data":"1a2073ff295233ad613b4c3c4b7c7cb4a51b9bc8f917b562bb3439f4b14ec3b5"} Sep 10 23:36:47.216209 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:47.216182 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dm6sw" event={"ID":"1da41227-eab5-43fa-a97c-69f7d935cc76","Type":"ContainerStarted","Data":"0f83e5908863fca26dd9df8811823cf20cbdcb3828521904d01f1d8ba1d71141"} Sep 10 23:36:47.217644 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:47.217622 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" event={"ID":"8df0f6be-ce76-4263-adfb-b1531272132e","Type":"ContainerStarted","Data":"b99ac536396cd5c78a6344558029fd3ada4f844aa08f154a059884fbe1004e5b"} Sep 10 23:36:47.224451 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:47.224405 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-f7fjr" podStartSLOduration=4.394879206 podStartE2EDuration="22.224393755s" totalImagesPullingTime="17.829514549s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:46.247980443 +0000 UTC m=+21.715454793" watchObservedRunningTime="2026-09-10 23:36:47.224393755 +0000 UTC m=+22.691868079" Sep 10 23:36:47.224838 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:47.224809 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-v77rf" podStartSLOduration=3.994639989 podStartE2EDuration="22.224802473s" totalImagesPullingTime="18.230162484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:47.224349683 +0000 UTC m=+22.691824008" watchObservedRunningTime="2026-09-10 23:36:47.224802473 +0000 UTC m=+22.692276797" Sep 10 23:36:47.239580 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:47.239506 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5pgnj" podStartSLOduration=4.384643725 podStartE2EDuration="22.239493858s" totalImagesPullingTime="17.854850133s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:47.239361561 +0000 UTC m=+22.706835886" watchObservedRunningTime="2026-09-10 23:36:47.239493858 +0000 UTC m=+22.706968196" Sep 10 23:36:47.252488 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:47.252454 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dm6sw" podStartSLOduration=4.430631601 podStartE2EDuration="22.252441814s" totalImagesPullingTime="17.821810213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:47.252079039 +0000 UTC m=+22.719553365" watchObservedRunningTime="2026-09-10 23:36:47.252441814 +0000 UTC m=+22.719916136" Sep 10 23:36:47.464634 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:47.464613 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 23:36:48.012873 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:48.012758 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T23:36:47.464629346Z","UUID":"f678f275-6523-40b5-bdd7-b90e85c4afd3","Handler":null,"Name":"","Endpoint":""} Sep 10 23:36:48.014607 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:48.014584 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 23:36:48.014736 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:48.014623 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 23:36:48.038496 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:48.038472 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:48.038641 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:48.038599 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmvxw" podUID="a06f8008-28ac-4cb9-8310-f5c42b169d10" Sep 10 23:36:48.222275 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:48.222036 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jfzt2" event={"ID":"3a055ce8-9e13-4aef-9d72-d14e198166a8","Type":"ContainerStarted","Data":"ccf970957ab0d547d8cd0ba9ee687c2f0fa790abba5a77f1ad3ce694b88a0403"} Sep 10 23:36:48.224748 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:48.224719 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" event={"ID":"446deb35-7fea-4d42-9fcb-c854037a9d31","Type":"ContainerStarted","Data":"ffbb7f78eae60078af49eb0a6f198ecb32c39187a3b463ce72aae18410f06dd6"} Sep 10 23:36:48.236325 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:48.236273 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jfzt2" podStartSLOduration=5.411290989 podStartE2EDuration="23.236256872s" totalImagesPullingTime="17.824965883s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:48.236000889 +0000 UTC m=+23.703475216" watchObservedRunningTime="2026-09-10 23:36:48.236256872 +0000 UTC m=+23.703731196" Sep 10 23:36:49.038981 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:49.038886 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:36:49.038981 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:49.038920 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:49.039217 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:49.039039 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fb4qx" podUID="c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d" Sep 10 23:36:49.039217 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:49.039160 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzfhc" podUID="9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7" Sep 10 23:36:49.228846 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:49.228808 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" event={"ID":"446deb35-7fea-4d42-9fcb-c854037a9d31","Type":"ContainerStarted","Data":"a803b4dfeafc95df445162c8e5ba362a9448c5ac1ada922aa176d61a4247fff9"} Sep 10 23:36:49.232810 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:49.232772 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" event={"ID":"a52bd1f0-6990-4ad6-a4fd-92499e4892a2","Type":"ContainerStarted","Data":"65eb55896ae1b77d0438e7b07a0fc6c6cf218e09e4979374081c00857d1f8046"} Sep 10 23:36:49.257087 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:49.257034 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npt56" podStartSLOduration=4.30801236 podStartE2EDuration="24.257014359s" totalImagesPullingTime="19.949001999s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:49.256463685 +0000 UTC m=+24.723938011" watchObservedRunningTime="2026-09-10 23:36:49.257014359 +0000 UTC m=+24.724488685" Sep 10 23:36:49.855263 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:49.855076 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kr8fb" Sep 10 23:36:49.855840 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:49.855820 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kr8fb" Sep 10 23:36:50.038181 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:50.038149 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:50.038355 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:50.038265 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmvxw" podUID="a06f8008-28ac-4cb9-8310-f5c42b169d10" Sep 10 23:36:50.235809 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:50.235732 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kr8fb" Sep 10 23:36:50.236302 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:50.236174 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kr8fb" Sep 10 23:36:51.038258 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:51.038225 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:36:51.038439 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:51.038298 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:51.038439 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:51.038397 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzfhc" podUID="9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7" Sep 10 23:36:51.038550 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:51.038492 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fb4qx" podUID="c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d" Sep 10 23:36:52.038896 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:52.038826 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:52.039637 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:52.038923 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmvxw" podUID="a06f8008-28ac-4cb9-8310-f5c42b169d10" Sep 10 23:36:52.241413 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:52.241385 2578 generic.go:356] "Generic (PLEG): container finished" podID="4580695f-ec27-45ad-9670-4849d9354b05" containerID="96aef25ce994671ff682215ed8a73ef8297750e63a87523a37d8f366ccb9441b" exitCode=0 Sep 10 23:36:52.241585 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:52.241478 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-whckt" event={"ID":"4580695f-ec27-45ad-9670-4849d9354b05","Type":"ContainerDied","Data":"96aef25ce994671ff682215ed8a73ef8297750e63a87523a37d8f366ccb9441b"} Sep 10 23:36:52.248542 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:52.248315 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" event={"ID":"a52bd1f0-6990-4ad6-a4fd-92499e4892a2","Type":"ContainerStarted","Data":"2f17a52e2af40b900b069e5eebbb3f3c8481533a081813bc730fca6faef110d2"} Sep 10 23:36:52.248688 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:52.248632 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:52.263358 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:52.263336 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:52.289684 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:52.289614 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" podStartSLOduration=9.40578098 podStartE2EDuration="27.289601466s" totalImagesPullingTime="17.883820486s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:52.289553723 +0000 UTC m=+27.757028047" watchObservedRunningTime="2026-09-10 23:36:52.289601466 +0000 UTC m=+27.757075785" Sep 10 23:36:53.038958 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:53.038919 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:53.039321 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:53.039027 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzfhc" podUID="9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7" Sep 10 23:36:53.039321 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:53.039113 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:36:53.039321 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:53.039192 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fb4qx" podUID="c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d" Sep 10 23:36:53.237207 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:53.237135 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zzfhc"] Sep 10 23:36:53.240957 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:53.240837 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fmvxw"] Sep 10 23:36:53.240957 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:53.240993 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:53.241223 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:53.241109 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmvxw" podUID="a06f8008-28ac-4cb9-8310-f5c42b169d10" Sep 10 23:36:53.241664 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:53.241640 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fb4qx"] Sep 10 23:36:53.252829 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:53.252807 2578 generic.go:356] "Generic (PLEG): container finished" podID="4580695f-ec27-45ad-9670-4849d9354b05" containerID="9b5006b6b196f2a16937e6462c089b232184469a2193360eb50cde2f28db6279" exitCode=0 Sep 10 23:36:53.252955 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:53.252916 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-whckt" event={"ID":"4580695f-ec27-45ad-9670-4849d9354b05","Type":"ContainerDied","Data":"9b5006b6b196f2a16937e6462c089b232184469a2193360eb50cde2f28db6279"} Sep 10 23:36:53.253307 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:53.253285 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:53.253412 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:53.253359 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:53.253412 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:53.253367 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:36:53.253520 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:53.253462 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:53.254193 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:53.253537 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzfhc" podUID="9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7" Sep 10 23:36:53.254193 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:53.253457 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fb4qx" podUID="c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d" Sep 10 23:36:53.269859 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:53.269838 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:36:54.256790 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:54.256705 2578 generic.go:356] "Generic (PLEG): container finished" podID="4580695f-ec27-45ad-9670-4849d9354b05" containerID="ea2139b20255e4d04006a574046fcfa486cc99f95f4fd94e8290767470f9b44d" exitCode=0 Sep 10 23:36:54.257159 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:54.256786 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-whckt" event={"ID":"4580695f-ec27-45ad-9670-4849d9354b05","Type":"ContainerDied","Data":"ea2139b20255e4d04006a574046fcfa486cc99f95f4fd94e8290767470f9b44d"} Sep 10 23:36:55.039290 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:55.039258 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:55.039435 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:55.039352 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:55.039435 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:55.039398 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:36:55.039435 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:55.039412 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmvxw" podUID="a06f8008-28ac-4cb9-8310-f5c42b169d10" Sep 10 23:36:55.039595 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:55.039471 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fb4qx" podUID="c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d" Sep 10 23:36:55.039595 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:55.039550 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzfhc" podUID="9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7" Sep 10 23:36:57.038547 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:57.038507 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:57.039003 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:57.038507 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:36:57.039003 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:57.038507 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:57.039003 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:57.038765 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fb4qx" podUID="c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d" Sep 10 23:36:57.039003 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:57.038649 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzfhc" podUID="9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7" Sep 10 23:36:57.039003 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:57.038862 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmvxw" podUID="a06f8008-28ac-4cb9-8310-f5c42b169d10" Sep 10 23:36:58.694925 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:58.694738 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-original-pull-secret\") pod \"global-pull-secret-syncer-zzfhc\" (UID: \"9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7\") " pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:58.695449 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:58.694894 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:58.695449 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:58.694979 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a06f8008-28ac-4cb9-8310-f5c42b169d10-metrics-certs\") pod \"network-metrics-daemon-fmvxw\" (UID: \"a06f8008-28ac-4cb9-8310-f5c42b169d10\") " pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:58.695449 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:58.695008 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-original-pull-secret podName:9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:30.694994312 +0000 UTC m=+66.162468619 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-original-pull-secret") pod "global-pull-secret-syncer-zzfhc" (UID: "9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7") : object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:58.695449 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:58.695072 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:58.695449 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:58.695123 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a06f8008-28ac-4cb9-8310-f5c42b169d10-metrics-certs podName:a06f8008-28ac-4cb9-8310-f5c42b169d10 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:30.695106691 +0000 UTC m=+66.162581012 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a06f8008-28ac-4cb9-8310-f5c42b169d10-metrics-certs") pod "network-metrics-daemon-fmvxw" (UID: "a06f8008-28ac-4cb9-8310-f5c42b169d10") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:58.896485 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:58.896444 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9drx\" (UniqueName: \"kubernetes.io/projected/c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d-kube-api-access-b9drx\") pod \"network-check-target-fb4qx\" (UID: \"c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d\") " pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:36:58.896646 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:58.896619 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 23:36:58.896646 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:58.896645 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 23:36:58.896750 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:58.896657 2578 projected.go:196] Error preparing data for projected volume kube-api-access-b9drx for pod openshift-network-diagnostics/network-check-target-fb4qx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:58.896750 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:58.896710 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d-kube-api-access-b9drx podName:c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d nodeName:}" failed. No retries permitted until 2026-09-10 23:37:30.896697212 +0000 UTC m=+66.364171515 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-b9drx" (UniqueName: "kubernetes.io/projected/c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d-kube-api-access-b9drx") pod "network-check-target-fb4qx" (UID: "c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:59.038285 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.038256 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:36:59.038402 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.038256 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:36:59.038402 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:59.038362 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzfhc" podUID="9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7" Sep 10 23:36:59.038510 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:59.038462 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fb4qx" podUID="c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d" Sep 10 23:36:59.038510 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.038256 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:36:59.038618 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:59.038586 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fmvxw" podUID="a06f8008-28ac-4cb9-8310-f5c42b169d10" Sep 10 23:36:59.271499 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.271415 2578 generic.go:356] "Generic (PLEG): container finished" podID="4580695f-ec27-45ad-9670-4849d9354b05" containerID="5fc87cab63f47ed7c03f79364d1a9014cdfe331dc753a43890cf701e3a085cdb" exitCode=0 Sep 10 23:36:59.271499 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.271453 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-whckt" event={"ID":"4580695f-ec27-45ad-9670-4849d9354b05","Type":"ContainerDied","Data":"5fc87cab63f47ed7c03f79364d1a9014cdfe331dc753a43890cf701e3a085cdb"} Sep 10 23:36:59.303761 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.303739 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-142.ec2.internal" event="NodeReady" Sep 10 23:36:59.303887 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.303875 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 23:36:59.340655 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.340628 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-677fbc4d-5cv5g"] Sep 10 23:36:59.364971 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.364928 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tmscj"] Sep 10 23:36:59.365066 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.365033 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:36:59.368184 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.368161 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-67vss\"" Sep 10 23:36:59.368330 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.368235 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 10 23:36:59.368579 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.368537 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 10 23:36:59.368830 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.368812 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 10 23:36:59.374470 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.374104 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 10 23:36:59.385059 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.385034 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-677fbc4d-5cv5g"] Sep 10 23:36:59.385155 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.385078 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-77hbh"] Sep 10 23:36:59.385214 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.385160 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tmscj" Sep 10 23:36:59.387564 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.387541 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 23:36:59.387564 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.387558 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 23:36:59.387701 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.387659 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6vpbd\"" Sep 10 23:36:59.404869 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.404850 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tmscj"] Sep 10 23:36:59.404972 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.404883 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-77hbh"] Sep 10 23:36:59.405022 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.404974 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-77hbh" Sep 10 23:36:59.407339 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.407318 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 23:36:59.407453 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.407318 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-45clb\"" Sep 10 23:36:59.407453 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.407322 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 23:36:59.407453 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.407327 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 23:36:59.500399 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.500373 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwxss\" (UniqueName: \"kubernetes.io/projected/0cb537d8-167e-4bcc-9681-1bdc99fd9cef-kube-api-access-fwxss\") pod \"ingress-canary-77hbh\" (UID: \"0cb537d8-167e-4bcc-9681-1bdc99fd9cef\") " pod="openshift-ingress-canary/ingress-canary-77hbh" Sep 10 23:36:59.500555 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.500403 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b74b357-db9d-4962-9eda-ff644ef6f739-tmp-dir\") pod \"dns-default-tmscj\" (UID: \"6b74b357-db9d-4962-9eda-ff644ef6f739\") " pod="openshift-dns/dns-default-tmscj" Sep 10 23:36:59.500555 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.500422 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/19629720-a2c6-4554-abe2-cc664a296910-registry-certificates\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:36:59.500555 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.500438 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/19629720-a2c6-4554-abe2-cc664a296910-installation-pull-secrets\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:36:59.500555 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.500482 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b74b357-db9d-4962-9eda-ff644ef6f739-metrics-tls\") pod \"dns-default-tmscj\" (UID: \"6b74b357-db9d-4962-9eda-ff644ef6f739\") " pod="openshift-dns/dns-default-tmscj" Sep 10 23:36:59.500555 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.500525 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6b74b357-db9d-4962-9eda-ff644ef6f739-config-volume\") pod \"dns-default-tmscj\" (UID: \"6b74b357-db9d-4962-9eda-ff644ef6f739\") " pod="openshift-dns/dns-default-tmscj" Sep 10 23:36:59.500811 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.500574 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtgwm\" (UniqueName: \"kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-kube-api-access-mtgwm\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:36:59.500811 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.500613 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/19629720-a2c6-4554-abe2-cc664a296910-image-registry-private-configuration\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:36:59.500811 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.500639 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-registry-tls\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:36:59.500811 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.500664 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j47mt\" (UniqueName: \"kubernetes.io/projected/6b74b357-db9d-4962-9eda-ff644ef6f739-kube-api-access-j47mt\") pod \"dns-default-tmscj\" (UID: \"6b74b357-db9d-4962-9eda-ff644ef6f739\") " pod="openshift-dns/dns-default-tmscj" Sep 10 23:36:59.500811 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.500703 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19629720-a2c6-4554-abe2-cc664a296910-trusted-ca\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:36:59.500811 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.500737 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-bound-sa-token\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:36:59.500811 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.500777 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/19629720-a2c6-4554-abe2-cc664a296910-ca-trust-extracted\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:36:59.500811 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.500801 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cb537d8-167e-4bcc-9681-1bdc99fd9cef-cert\") pod \"ingress-canary-77hbh\" (UID: \"0cb537d8-167e-4bcc-9681-1bdc99fd9cef\") " pod="openshift-ingress-canary/ingress-canary-77hbh" Sep 10 23:36:59.514820 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.514796 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kr8fb_4b358463-cfe2-49ec-91ea-64ae0856ccd7/konnectivity-agent/0.log" limit=1024 Sep 10 23:36:59.601248 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.601164 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-bound-sa-token\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:36:59.601248 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.601217 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/19629720-a2c6-4554-abe2-cc664a296910-ca-trust-extracted\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:36:59.601248 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.601239 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cb537d8-167e-4bcc-9681-1bdc99fd9cef-cert\") pod \"ingress-canary-77hbh\" (UID: \"0cb537d8-167e-4bcc-9681-1bdc99fd9cef\") " pod="openshift-ingress-canary/ingress-canary-77hbh" Sep 10 23:36:59.601498 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.601273 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwxss\" (UniqueName: \"kubernetes.io/projected/0cb537d8-167e-4bcc-9681-1bdc99fd9cef-kube-api-access-fwxss\") pod \"ingress-canary-77hbh\" (UID: \"0cb537d8-167e-4bcc-9681-1bdc99fd9cef\") " pod="openshift-ingress-canary/ingress-canary-77hbh" Sep 10 23:36:59.601498 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.601292 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b74b357-db9d-4962-9eda-ff644ef6f739-tmp-dir\") pod \"dns-default-tmscj\" (UID: \"6b74b357-db9d-4962-9eda-ff644ef6f739\") " pod="openshift-dns/dns-default-tmscj" Sep 10 23:36:59.601498 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.601312 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/19629720-a2c6-4554-abe2-cc664a296910-registry-certificates\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:36:59.601498 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.601331 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/19629720-a2c6-4554-abe2-cc664a296910-installation-pull-secrets\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:36:59.601498 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.601351 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b74b357-db9d-4962-9eda-ff644ef6f739-metrics-tls\") pod \"dns-default-tmscj\" (UID: \"6b74b357-db9d-4962-9eda-ff644ef6f739\") " pod="openshift-dns/dns-default-tmscj" Sep 10 23:36:59.601498 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.601376 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6b74b357-db9d-4962-9eda-ff644ef6f739-config-volume\") pod \"dns-default-tmscj\" (UID: \"6b74b357-db9d-4962-9eda-ff644ef6f739\") " pod="openshift-dns/dns-default-tmscj" Sep 10 23:36:59.601498 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:59.601495 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 23:36:59.601838 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:59.601571 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b74b357-db9d-4962-9eda-ff644ef6f739-metrics-tls podName:6b74b357-db9d-4962-9eda-ff644ef6f739 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:00.101552508 +0000 UTC m=+35.569026831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b74b357-db9d-4962-9eda-ff644ef6f739-metrics-tls") pod "dns-default-tmscj" (UID: "6b74b357-db9d-4962-9eda-ff644ef6f739") : secret "dns-default-metrics-tls" not found Sep 10 23:36:59.601838 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.601608 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtgwm\" (UniqueName: \"kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-kube-api-access-mtgwm\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:36:59.601838 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.601653 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/19629720-a2c6-4554-abe2-cc664a296910-image-registry-private-configuration\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:36:59.601838 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.601678 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-registry-tls\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:36:59.601838 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.601676 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b74b357-db9d-4962-9eda-ff644ef6f739-tmp-dir\") pod \"dns-default-tmscj\" (UID: \"6b74b357-db9d-4962-9eda-ff644ef6f739\") " pod="openshift-dns/dns-default-tmscj" Sep 10 23:36:59.601838 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.601703 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j47mt\" (UniqueName: \"kubernetes.io/projected/6b74b357-db9d-4962-9eda-ff644ef6f739-kube-api-access-j47mt\") pod \"dns-default-tmscj\" (UID: \"6b74b357-db9d-4962-9eda-ff644ef6f739\") " pod="openshift-dns/dns-default-tmscj" Sep 10 23:36:59.601838 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.601744 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/19629720-a2c6-4554-abe2-cc664a296910-ca-trust-extracted\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:36:59.601838 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.601731 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19629720-a2c6-4554-abe2-cc664a296910-trusted-ca\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:36:59.602317 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:59.601923 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 23:36:59.602317 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:59.601956 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-677fbc4d-5cv5g: secret "image-registry-tls" not found Sep 10 23:36:59.602317 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:36:59.602014 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-registry-tls podName:19629720-a2c6-4554-abe2-cc664a296910 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:00.101995909 +0000 UTC m=+35.569470231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-registry-tls") pod "image-registry-677fbc4d-5cv5g" (UID: "19629720-a2c6-4554-abe2-cc664a296910") : secret "image-registry-tls" not found Sep 10 23:36:59.602317 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.602068 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/19629720-a2c6-4554-abe2-cc664a296910-registry-certificates\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:36:59.602523 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.602478 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6b74b357-db9d-4962-9eda-ff644ef6f739-config-volume\") pod \"dns-default-tmscj\" (UID: \"6b74b357-db9d-4962-9eda-ff644ef6f739\") " pod="openshift-dns/dns-default-tmscj" Sep 10 23:36:59.602833 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.602814 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19629720-a2c6-4554-abe2-cc664a296910-trusted-ca\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:36:59.605068 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.605044 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kr8fb_4b358463-cfe2-49ec-91ea-64ae0856ccd7/konnectivity-agent/0.log" limit=1024 Sep 10 23:36:59.605596 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.605574 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/19629720-a2c6-4554-abe2-cc664a296910-image-registry-private-configuration\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:36:59.605596 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.605587 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cb537d8-167e-4bcc-9681-1bdc99fd9cef-cert\") pod \"ingress-canary-77hbh\" (UID: \"0cb537d8-167e-4bcc-9681-1bdc99fd9cef\") " pod="openshift-ingress-canary/ingress-canary-77hbh" Sep 10 23:36:59.605750 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.605612 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/19629720-a2c6-4554-abe2-cc664a296910-installation-pull-secrets\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:36:59.612422 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.612391 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-bound-sa-token\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:36:59.612533 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.612510 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwxss\" (UniqueName: \"kubernetes.io/projected/0cb537d8-167e-4bcc-9681-1bdc99fd9cef-kube-api-access-fwxss\") pod \"ingress-canary-77hbh\" (UID: \"0cb537d8-167e-4bcc-9681-1bdc99fd9cef\") " pod="openshift-ingress-canary/ingress-canary-77hbh" Sep 10 23:36:59.612678 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.612662 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtgwm\" (UniqueName: \"kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-kube-api-access-mtgwm\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:36:59.612767 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.612749 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j47mt\" (UniqueName: \"kubernetes.io/projected/6b74b357-db9d-4962-9eda-ff644ef6f739-kube-api-access-j47mt\") pod \"dns-default-tmscj\" (UID: \"6b74b357-db9d-4962-9eda-ff644ef6f739\") " pod="openshift-dns/dns-default-tmscj" Sep 10 23:36:59.713225 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:36:59.713192 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-77hbh" Sep 10 23:37:00.024378 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:00.024308 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-77hbh"] Sep 10 23:37:00.028306 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:37:00.028278 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb537d8_167e_4bcc_9681_1bdc99fd9cef.slice/crio-ca4d9b70bef734e16cedce9b35a2a61ed17bd8e42f11bcd58454a06d28e37ef0 WatchSource:0}: Error finding container ca4d9b70bef734e16cedce9b35a2a61ed17bd8e42f11bcd58454a06d28e37ef0: Status 404 returned error can't find the container with id ca4d9b70bef734e16cedce9b35a2a61ed17bd8e42f11bcd58454a06d28e37ef0 Sep 10 23:37:00.105210 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:00.105177 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b74b357-db9d-4962-9eda-ff644ef6f739-metrics-tls\") pod \"dns-default-tmscj\" (UID: \"6b74b357-db9d-4962-9eda-ff644ef6f739\") " pod="openshift-dns/dns-default-tmscj" Sep 10 23:37:00.105399 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:00.105240 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-registry-tls\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:37:00.105399 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:00.105320 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 23:37:00.105399 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:00.105337 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 23:37:00.105399 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:00.105351 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-677fbc4d-5cv5g: secret "image-registry-tls" not found Sep 10 23:37:00.105399 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:00.105401 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b74b357-db9d-4962-9eda-ff644ef6f739-metrics-tls podName:6b74b357-db9d-4962-9eda-ff644ef6f739 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:01.105384476 +0000 UTC m=+36.572858783 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b74b357-db9d-4962-9eda-ff644ef6f739-metrics-tls") pod "dns-default-tmscj" (UID: "6b74b357-db9d-4962-9eda-ff644ef6f739") : secret "dns-default-metrics-tls" not found Sep 10 23:37:00.105594 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:00.105416 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-registry-tls podName:19629720-a2c6-4554-abe2-cc664a296910 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:01.105409494 +0000 UTC m=+36.572883796 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-registry-tls") pod "image-registry-677fbc4d-5cv5g" (UID: "19629720-a2c6-4554-abe2-cc664a296910") : secret "image-registry-tls" not found Sep 10 23:37:00.276531 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:00.276460 2578 generic.go:356] "Generic (PLEG): container finished" podID="4580695f-ec27-45ad-9670-4849d9354b05" containerID="c6bdd0509d3657369e462319c4ca08d25784f251f00813eb1c507979984face1" exitCode=0 Sep 10 23:37:00.276531 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:00.276516 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-whckt" event={"ID":"4580695f-ec27-45ad-9670-4849d9354b05","Type":"ContainerDied","Data":"c6bdd0509d3657369e462319c4ca08d25784f251f00813eb1c507979984face1"} Sep 10 23:37:00.277730 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:00.277707 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-77hbh" event={"ID":"0cb537d8-167e-4bcc-9681-1bdc99fd9cef","Type":"ContainerStarted","Data":"ca4d9b70bef734e16cedce9b35a2a61ed17bd8e42f11bcd58454a06d28e37ef0"} Sep 10 23:37:01.038514 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:01.038478 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:37:01.039329 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:01.038484 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:37:01.039329 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:01.038612 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:37:01.041886 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:01.041865 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cql7l\"" Sep 10 23:37:01.042014 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:01.041866 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 23:37:01.042014 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:01.041866 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xgqvn\"" Sep 10 23:37:01.042231 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:01.042213 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 23:37:01.042345 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:01.042247 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 23:37:01.042345 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:01.042298 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 23:37:01.112983 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:01.112958 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b74b357-db9d-4962-9eda-ff644ef6f739-metrics-tls\") pod \"dns-default-tmscj\" (UID: \"6b74b357-db9d-4962-9eda-ff644ef6f739\") " pod="openshift-dns/dns-default-tmscj" Sep 10 23:37:01.113115 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:01.113022 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-registry-tls\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:37:01.113115 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:01.113105 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 23:37:01.113220 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:01.113150 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 23:37:01.113220 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:01.113165 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-677fbc4d-5cv5g: secret "image-registry-tls" not found Sep 10 23:37:01.113220 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:01.113175 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b74b357-db9d-4962-9eda-ff644ef6f739-metrics-tls podName:6b74b357-db9d-4962-9eda-ff644ef6f739 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:03.113156539 +0000 UTC m=+38.580630857 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b74b357-db9d-4962-9eda-ff644ef6f739-metrics-tls") pod "dns-default-tmscj" (UID: "6b74b357-db9d-4962-9eda-ff644ef6f739") : secret "dns-default-metrics-tls" not found Sep 10 23:37:01.113220 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:01.113199 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-registry-tls podName:19629720-a2c6-4554-abe2-cc664a296910 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:03.113188138 +0000 UTC m=+38.580662458 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-registry-tls") pod "image-registry-677fbc4d-5cv5g" (UID: "19629720-a2c6-4554-abe2-cc664a296910") : secret "image-registry-tls" not found Sep 10 23:37:01.284662 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:01.284624 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-whckt" event={"ID":"4580695f-ec27-45ad-9670-4849d9354b05","Type":"ContainerStarted","Data":"e06334b475873bfa5d558b3a7ae1dabc73a02f1e5fa1397ec01df049863b0be8"} Sep 10 23:37:01.303627 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:01.303519 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-whckt" podStartSLOduration=8.53218259 podStartE2EDuration="36.303497004s" totalImagesPullingTime="27.771314414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:01.303340451 +0000 UTC m=+36.770814793" watchObservedRunningTime="2026-09-10 23:37:01.303497004 +0000 UTC m=+36.770971329" Sep 10 23:37:01.425097 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:01.425059 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kr8fb_4b358463-cfe2-49ec-91ea-64ae0856ccd7/konnectivity-agent/0.log" limit=1024 Sep 10 23:37:01.482074 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:01.482029 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kr8fb_4b358463-cfe2-49ec-91ea-64ae0856ccd7/konnectivity-agent/0.log" limit=1024 Sep 10 23:37:03.129055 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:03.128977 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-registry-tls\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:37:03.129055 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:03.129053 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b74b357-db9d-4962-9eda-ff644ef6f739-metrics-tls\") pod \"dns-default-tmscj\" (UID: \"6b74b357-db9d-4962-9eda-ff644ef6f739\") " pod="openshift-dns/dns-default-tmscj" Sep 10 23:37:03.129401 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:03.129139 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 23:37:03.129401 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:03.129163 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-677fbc4d-5cv5g: secret "image-registry-tls" not found Sep 10 23:37:03.129401 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:03.129212 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-registry-tls podName:19629720-a2c6-4554-abe2-cc664a296910 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:07.129196771 +0000 UTC m=+42.596671074 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-registry-tls") pod "image-registry-677fbc4d-5cv5g" (UID: "19629720-a2c6-4554-abe2-cc664a296910") : secret "image-registry-tls" not found Sep 10 23:37:03.129401 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:03.129143 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 23:37:03.129401 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:03.129273 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b74b357-db9d-4962-9eda-ff644ef6f739-metrics-tls podName:6b74b357-db9d-4962-9eda-ff644ef6f739 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:07.129260845 +0000 UTC m=+42.596735148 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b74b357-db9d-4962-9eda-ff644ef6f739-metrics-tls") pod "dns-default-tmscj" (UID: "6b74b357-db9d-4962-9eda-ff644ef6f739") : secret "dns-default-metrics-tls" not found Sep 10 23:37:03.291421 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:03.291391 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-77hbh" event={"ID":"0cb537d8-167e-4bcc-9681-1bdc99fd9cef","Type":"ContainerStarted","Data":"ae8fbe2d3a87a83a55927bf420992020caf8629bf674b0efae2828dd37156f3e"} Sep 10 23:37:03.310785 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:03.310734 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-77hbh" podStartSLOduration=1.93496968 podStartE2EDuration="4.310719047s" totalImagesPullingTime="2.375749367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:03.309432067 +0000 UTC m=+38.776906392" watchObservedRunningTime="2026-09-10 23:37:03.310719047 +0000 UTC m=+38.778193371" Sep 10 23:37:07.159184 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:07.159143 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b74b357-db9d-4962-9eda-ff644ef6f739-metrics-tls\") pod \"dns-default-tmscj\" (UID: \"6b74b357-db9d-4962-9eda-ff644ef6f739\") " pod="openshift-dns/dns-default-tmscj" Sep 10 23:37:07.159566 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:07.159205 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-registry-tls\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:37:07.159566 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:07.159301 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 23:37:07.159566 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:07.159364 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b74b357-db9d-4962-9eda-ff644ef6f739-metrics-tls podName:6b74b357-db9d-4962-9eda-ff644ef6f739 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:15.159349891 +0000 UTC m=+50.626824197 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b74b357-db9d-4962-9eda-ff644ef6f739-metrics-tls") pod "dns-default-tmscj" (UID: "6b74b357-db9d-4962-9eda-ff644ef6f739") : secret "dns-default-metrics-tls" not found Sep 10 23:37:07.159566 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:07.159304 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 23:37:07.159566 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:07.159410 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-677fbc4d-5cv5g: secret "image-registry-tls" not found Sep 10 23:37:07.159566 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:07.159446 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-registry-tls podName:19629720-a2c6-4554-abe2-cc664a296910 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:15.159434924 +0000 UTC m=+50.626909233 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-registry-tls") pod "image-registry-677fbc4d-5cv5g" (UID: "19629720-a2c6-4554-abe2-cc664a296910") : secret "image-registry-tls" not found Sep 10 23:37:15.218407 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:15.218365 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b74b357-db9d-4962-9eda-ff644ef6f739-metrics-tls\") pod \"dns-default-tmscj\" (UID: \"6b74b357-db9d-4962-9eda-ff644ef6f739\") " pod="openshift-dns/dns-default-tmscj" Sep 10 23:37:15.218815 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:15.218423 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-registry-tls\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:37:15.218815 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:15.218509 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 23:37:15.218815 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:15.218534 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 23:37:15.218815 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:15.218544 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-677fbc4d-5cv5g: secret "image-registry-tls" not found Sep 10 23:37:15.218815 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:15.218582 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b74b357-db9d-4962-9eda-ff644ef6f739-metrics-tls podName:6b74b357-db9d-4962-9eda-ff644ef6f739 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:31.218566437 +0000 UTC m=+66.686040739 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b74b357-db9d-4962-9eda-ff644ef6f739-metrics-tls") pod "dns-default-tmscj" (UID: "6b74b357-db9d-4962-9eda-ff644ef6f739") : secret "dns-default-metrics-tls" not found Sep 10 23:37:15.218815 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:15.218598 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-registry-tls podName:19629720-a2c6-4554-abe2-cc664a296910 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:31.218591132 +0000 UTC m=+66.686065435 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-registry-tls") pod "image-registry-677fbc4d-5cv5g" (UID: "19629720-a2c6-4554-abe2-cc664a296910") : secret "image-registry-tls" not found Sep 10 23:37:25.268688 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:25.268661 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8srbc" Sep 10 23:37:27.465951 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.465910 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-zkr6f"] Sep 10 23:37:27.468570 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.468553 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" Sep 10 23:37:27.471001 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.470979 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ff5g5\"" Sep 10 23:37:27.471118 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.471030 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 10 23:37:27.471970 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.471929 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 10 23:37:27.472083 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.472010 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 10 23:37:27.472083 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.472018 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 10 23:37:27.477741 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.477714 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 10 23:37:27.479146 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.479125 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-zkr6f"] Sep 10 23:37:27.505529 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.505495 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aacd921b-5c15-4f7f-8685-cfceebfe5407-config\") pod \"console-operator-696884fb9c-zkr6f\" (UID: \"aacd921b-5c15-4f7f-8685-cfceebfe5407\") " pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" Sep 10 23:37:27.505675 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.505547 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aacd921b-5c15-4f7f-8685-cfceebfe5407-trusted-ca\") pod \"console-operator-696884fb9c-zkr6f\" (UID: \"aacd921b-5c15-4f7f-8685-cfceebfe5407\") " pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" Sep 10 23:37:27.505675 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.505572 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aacd921b-5c15-4f7f-8685-cfceebfe5407-serving-cert\") pod \"console-operator-696884fb9c-zkr6f\" (UID: \"aacd921b-5c15-4f7f-8685-cfceebfe5407\") " pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" Sep 10 23:37:27.505675 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.505640 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrfpl\" (UniqueName: \"kubernetes.io/projected/aacd921b-5c15-4f7f-8685-cfceebfe5407-kube-api-access-wrfpl\") pod \"console-operator-696884fb9c-zkr6f\" (UID: \"aacd921b-5c15-4f7f-8685-cfceebfe5407\") " pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" Sep 10 23:37:27.561064 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.561028 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-gbhmg"] Sep 10 23:37:27.562789 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.562774 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-gbhmg" Sep 10 23:37:27.566145 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.566121 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-whpjq\"" Sep 10 23:37:27.566279 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.566157 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 10 23:37:27.566279 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.566214 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 10 23:37:27.566693 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.566675 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 23:37:27.568478 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.568458 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 23:37:27.572724 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.572706 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 10 23:37:27.578321 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.578297 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-gbhmg"] Sep 10 23:37:27.606500 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.606470 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aacd921b-5c15-4f7f-8685-cfceebfe5407-config\") pod \"console-operator-696884fb9c-zkr6f\" (UID: \"aacd921b-5c15-4f7f-8685-cfceebfe5407\") " pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" Sep 10 23:37:27.606500 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.606506 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b3a36399-90fc-4dbe-86f4-ffc633cd89cc-snapshots\") pod \"insights-operator-5844b856db-gbhmg\" (UID: \"b3a36399-90fc-4dbe-86f4-ffc633cd89cc\") " pod="openshift-insights/insights-operator-5844b856db-gbhmg" Sep 10 23:37:27.606725 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.606550 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aacd921b-5c15-4f7f-8685-cfceebfe5407-trusted-ca\") pod \"console-operator-696884fb9c-zkr6f\" (UID: \"aacd921b-5c15-4f7f-8685-cfceebfe5407\") " pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" Sep 10 23:37:27.606725 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.606587 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aacd921b-5c15-4f7f-8685-cfceebfe5407-serving-cert\") pod \"console-operator-696884fb9c-zkr6f\" (UID: \"aacd921b-5c15-4f7f-8685-cfceebfe5407\") " pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" Sep 10 23:37:27.606725 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.606610 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8pln\" (UniqueName: \"kubernetes.io/projected/b3a36399-90fc-4dbe-86f4-ffc633cd89cc-kube-api-access-w8pln\") pod \"insights-operator-5844b856db-gbhmg\" (UID: \"b3a36399-90fc-4dbe-86f4-ffc633cd89cc\") " pod="openshift-insights/insights-operator-5844b856db-gbhmg" Sep 10 23:37:27.606725 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.606694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrfpl\" (UniqueName: \"kubernetes.io/projected/aacd921b-5c15-4f7f-8685-cfceebfe5407-kube-api-access-wrfpl\") pod \"console-operator-696884fb9c-zkr6f\" (UID: \"aacd921b-5c15-4f7f-8685-cfceebfe5407\") " pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" Sep 10 23:37:27.606725 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.606726 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3a36399-90fc-4dbe-86f4-ffc633cd89cc-tmp\") pod \"insights-operator-5844b856db-gbhmg\" (UID: \"b3a36399-90fc-4dbe-86f4-ffc633cd89cc\") " pod="openshift-insights/insights-operator-5844b856db-gbhmg" Sep 10 23:37:27.606931 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.606747 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3a36399-90fc-4dbe-86f4-ffc633cd89cc-trusted-ca-bundle\") pod \"insights-operator-5844b856db-gbhmg\" (UID: \"b3a36399-90fc-4dbe-86f4-ffc633cd89cc\") " pod="openshift-insights/insights-operator-5844b856db-gbhmg" Sep 10 23:37:27.606931 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.606772 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3a36399-90fc-4dbe-86f4-ffc633cd89cc-service-ca-bundle\") pod \"insights-operator-5844b856db-gbhmg\" (UID: \"b3a36399-90fc-4dbe-86f4-ffc633cd89cc\") " pod="openshift-insights/insights-operator-5844b856db-gbhmg" Sep 10 23:37:27.606931 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.606812 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3a36399-90fc-4dbe-86f4-ffc633cd89cc-serving-cert\") pod \"insights-operator-5844b856db-gbhmg\" (UID: \"b3a36399-90fc-4dbe-86f4-ffc633cd89cc\") " pod="openshift-insights/insights-operator-5844b856db-gbhmg" Sep 10 23:37:27.607293 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.607268 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aacd921b-5c15-4f7f-8685-cfceebfe5407-config\") pod \"console-operator-696884fb9c-zkr6f\" (UID: \"aacd921b-5c15-4f7f-8685-cfceebfe5407\") " pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" Sep 10 23:37:27.607415 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.607399 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aacd921b-5c15-4f7f-8685-cfceebfe5407-trusted-ca\") pod \"console-operator-696884fb9c-zkr6f\" (UID: \"aacd921b-5c15-4f7f-8685-cfceebfe5407\") " pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" Sep 10 23:37:27.610147 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.610127 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aacd921b-5c15-4f7f-8685-cfceebfe5407-serving-cert\") pod \"console-operator-696884fb9c-zkr6f\" (UID: \"aacd921b-5c15-4f7f-8685-cfceebfe5407\") " pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" Sep 10 23:37:27.615623 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.615604 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrfpl\" (UniqueName: \"kubernetes.io/projected/aacd921b-5c15-4f7f-8685-cfceebfe5407-kube-api-access-wrfpl\") pod \"console-operator-696884fb9c-zkr6f\" (UID: \"aacd921b-5c15-4f7f-8685-cfceebfe5407\") " pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" Sep 10 23:37:27.707760 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.707714 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8pln\" (UniqueName: \"kubernetes.io/projected/b3a36399-90fc-4dbe-86f4-ffc633cd89cc-kube-api-access-w8pln\") pod \"insights-operator-5844b856db-gbhmg\" (UID: \"b3a36399-90fc-4dbe-86f4-ffc633cd89cc\") " pod="openshift-insights/insights-operator-5844b856db-gbhmg" Sep 10 23:37:27.707979 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.707806 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3a36399-90fc-4dbe-86f4-ffc633cd89cc-tmp\") pod \"insights-operator-5844b856db-gbhmg\" (UID: \"b3a36399-90fc-4dbe-86f4-ffc633cd89cc\") " pod="openshift-insights/insights-operator-5844b856db-gbhmg" Sep 10 23:37:27.707979 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.707837 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3a36399-90fc-4dbe-86f4-ffc633cd89cc-trusted-ca-bundle\") pod \"insights-operator-5844b856db-gbhmg\" (UID: \"b3a36399-90fc-4dbe-86f4-ffc633cd89cc\") " pod="openshift-insights/insights-operator-5844b856db-gbhmg" Sep 10 23:37:27.707979 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.707861 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3a36399-90fc-4dbe-86f4-ffc633cd89cc-service-ca-bundle\") pod \"insights-operator-5844b856db-gbhmg\" (UID: \"b3a36399-90fc-4dbe-86f4-ffc633cd89cc\") " pod="openshift-insights/insights-operator-5844b856db-gbhmg" Sep 10 23:37:27.707979 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.707920 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3a36399-90fc-4dbe-86f4-ffc633cd89cc-serving-cert\") pod \"insights-operator-5844b856db-gbhmg\" (UID: \"b3a36399-90fc-4dbe-86f4-ffc633cd89cc\") " pod="openshift-insights/insights-operator-5844b856db-gbhmg" Sep 10 23:37:27.708228 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.708017 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b3a36399-90fc-4dbe-86f4-ffc633cd89cc-snapshots\") pod \"insights-operator-5844b856db-gbhmg\" (UID: \"b3a36399-90fc-4dbe-86f4-ffc633cd89cc\") " pod="openshift-insights/insights-operator-5844b856db-gbhmg" Sep 10 23:37:27.708299 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.708277 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3a36399-90fc-4dbe-86f4-ffc633cd89cc-tmp\") pod \"insights-operator-5844b856db-gbhmg\" (UID: \"b3a36399-90fc-4dbe-86f4-ffc633cd89cc\") " pod="openshift-insights/insights-operator-5844b856db-gbhmg" Sep 10 23:37:27.708467 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.708446 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3a36399-90fc-4dbe-86f4-ffc633cd89cc-service-ca-bundle\") pod \"insights-operator-5844b856db-gbhmg\" (UID: \"b3a36399-90fc-4dbe-86f4-ffc633cd89cc\") " pod="openshift-insights/insights-operator-5844b856db-gbhmg" Sep 10 23:37:27.708548 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.708501 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b3a36399-90fc-4dbe-86f4-ffc633cd89cc-snapshots\") pod \"insights-operator-5844b856db-gbhmg\" (UID: \"b3a36399-90fc-4dbe-86f4-ffc633cd89cc\") " pod="openshift-insights/insights-operator-5844b856db-gbhmg" Sep 10 23:37:27.708740 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.708718 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3a36399-90fc-4dbe-86f4-ffc633cd89cc-trusted-ca-bundle\") pod \"insights-operator-5844b856db-gbhmg\" (UID: \"b3a36399-90fc-4dbe-86f4-ffc633cd89cc\") " pod="openshift-insights/insights-operator-5844b856db-gbhmg" Sep 10 23:37:27.710216 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.710197 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3a36399-90fc-4dbe-86f4-ffc633cd89cc-serving-cert\") pod \"insights-operator-5844b856db-gbhmg\" (UID: \"b3a36399-90fc-4dbe-86f4-ffc633cd89cc\") " pod="openshift-insights/insights-operator-5844b856db-gbhmg" Sep 10 23:37:27.715807 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.715785 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8pln\" (UniqueName: \"kubernetes.io/projected/b3a36399-90fc-4dbe-86f4-ffc633cd89cc-kube-api-access-w8pln\") pod \"insights-operator-5844b856db-gbhmg\" (UID: \"b3a36399-90fc-4dbe-86f4-ffc633cd89cc\") " pod="openshift-insights/insights-operator-5844b856db-gbhmg" Sep 10 23:37:27.780127 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.780036 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" Sep 10 23:37:27.872314 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:27.872278 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-gbhmg" Sep 10 23:37:28.047073 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:28.046995 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-zkr6f"] Sep 10 23:37:28.049948 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:37:28.049899 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaacd921b_5c15_4f7f_8685_cfceebfe5407.slice/crio-bd97a370a8cd309d919211d2f9736b88421b1acb3b41bef235db40a1142b0c88 WatchSource:0}: Error finding container bd97a370a8cd309d919211d2f9736b88421b1acb3b41bef235db40a1142b0c88: Status 404 returned error can't find the container with id bd97a370a8cd309d919211d2f9736b88421b1acb3b41bef235db40a1142b0c88 Sep 10 23:37:28.136159 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:28.136130 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-gbhmg"] Sep 10 23:37:28.139079 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:37:28.139055 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a36399_90fc_4dbe_86f4_ffc633cd89cc.slice/crio-043384cc6e2d478fcabcda31446051efc40c105e58a90b3b445bf67fee93984a WatchSource:0}: Error finding container 043384cc6e2d478fcabcda31446051efc40c105e58a90b3b445bf67fee93984a: Status 404 returned error can't find the container with id 043384cc6e2d478fcabcda31446051efc40c105e58a90b3b445bf67fee93984a Sep 10 23:37:28.351050 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:28.350965 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-gbhmg" event={"ID":"b3a36399-90fc-4dbe-86f4-ffc633cd89cc","Type":"ContainerStarted","Data":"043384cc6e2d478fcabcda31446051efc40c105e58a90b3b445bf67fee93984a"} Sep 10 23:37:28.351984 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:28.351957 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" event={"ID":"aacd921b-5c15-4f7f-8685-cfceebfe5407","Type":"ContainerStarted","Data":"bd97a370a8cd309d919211d2f9736b88421b1acb3b41bef235db40a1142b0c88"} Sep 10 23:37:30.733456 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:30.733423 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a06f8008-28ac-4cb9-8310-f5c42b169d10-metrics-certs\") pod \"network-metrics-daemon-fmvxw\" (UID: \"a06f8008-28ac-4cb9-8310-f5c42b169d10\") " pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:37:30.733780 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:30.733593 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-original-pull-secret\") pod \"global-pull-secret-syncer-zzfhc\" (UID: \"9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7\") " pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:37:30.736265 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:30.736240 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 23:37:30.736392 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:30.736279 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 23:37:30.744335 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:30.744320 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 10 23:37:30.744410 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:30.744373 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a06f8008-28ac-4cb9-8310-f5c42b169d10-metrics-certs podName:a06f8008-28ac-4cb9-8310-f5c42b169d10 nodeName:}" failed. No retries permitted until 2026-09-10 23:38:34.744357775 +0000 UTC m=+130.211832078 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a06f8008-28ac-4cb9-8310-f5c42b169d10-metrics-certs") pod "network-metrics-daemon-fmvxw" (UID: "a06f8008-28ac-4cb9-8310-f5c42b169d10") : secret "metrics-daemon-secret" not found Sep 10 23:37:30.746334 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:30.746311 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7-original-pull-secret\") pod \"global-pull-secret-syncer-zzfhc\" (UID: \"9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7\") " pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:37:30.757483 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:30.757464 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzfhc" Sep 10 23:37:30.934488 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:30.934452 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9drx\" (UniqueName: \"kubernetes.io/projected/c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d-kube-api-access-b9drx\") pod \"network-check-target-fb4qx\" (UID: \"c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d\") " pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:37:30.937281 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:30.937260 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 23:37:30.952440 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:30.951968 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 23:37:30.961047 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:30.961026 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9drx\" (UniqueName: \"kubernetes.io/projected/c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d-kube-api-access-b9drx\") pod \"network-check-target-fb4qx\" (UID: \"c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d\") " pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:37:31.052316 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:31.052288 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xgqvn\"" Sep 10 23:37:31.060746 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:31.060708 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:37:31.171328 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:31.171289 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zzfhc"] Sep 10 23:37:31.174469 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:37:31.174445 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9944f5db_ef8d_4f9a_86a8_a7c4a3838cb7.slice/crio-86cb502dd6fdb501d60a466e94978525daa838491855bc22645856ed9bc4fd9b WatchSource:0}: Error finding container 86cb502dd6fdb501d60a466e94978525daa838491855bc22645856ed9bc4fd9b: Status 404 returned error can't find the container with id 86cb502dd6fdb501d60a466e94978525daa838491855bc22645856ed9bc4fd9b Sep 10 23:37:31.237664 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:31.237579 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-registry-tls\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:37:31.237791 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:31.237667 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b74b357-db9d-4962-9eda-ff644ef6f739-metrics-tls\") pod \"dns-default-tmscj\" (UID: \"6b74b357-db9d-4962-9eda-ff644ef6f739\") " pod="openshift-dns/dns-default-tmscj" Sep 10 23:37:31.237791 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:31.237735 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 23:37:31.237791 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:31.237756 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-677fbc4d-5cv5g: secret "image-registry-tls" not found Sep 10 23:37:31.237897 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:31.237799 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 23:37:31.237897 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:31.237818 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-registry-tls podName:19629720-a2c6-4554-abe2-cc664a296910 nodeName:}" failed. No retries permitted until 2026-09-10 23:38:03.237797752 +0000 UTC m=+98.705272066 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-registry-tls") pod "image-registry-677fbc4d-5cv5g" (UID: "19629720-a2c6-4554-abe2-cc664a296910") : secret "image-registry-tls" not found Sep 10 23:37:31.237897 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:31.237847 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b74b357-db9d-4962-9eda-ff644ef6f739-metrics-tls podName:6b74b357-db9d-4962-9eda-ff644ef6f739 nodeName:}" failed. No retries permitted until 2026-09-10 23:38:03.2378321 +0000 UTC m=+98.705306405 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b74b357-db9d-4962-9eda-ff644ef6f739-metrics-tls") pod "dns-default-tmscj" (UID: "6b74b357-db9d-4962-9eda-ff644ef6f739") : secret "dns-default-metrics-tls" not found Sep 10 23:37:31.346166 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:31.346131 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fb4qx"] Sep 10 23:37:31.348828 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:37:31.348799 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3fb908c_f83c_4f4d_bdc1_2ad529ae5e8d.slice/crio-7fcf7020f1d660bb0a521c85b0d2c5351c766a65de67bcd05d336b8ca4b0d66f WatchSource:0}: Error finding container 7fcf7020f1d660bb0a521c85b0d2c5351c766a65de67bcd05d336b8ca4b0d66f: Status 404 returned error can't find the container with id 7fcf7020f1d660bb0a521c85b0d2c5351c766a65de67bcd05d336b8ca4b0d66f Sep 10 23:37:31.362293 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:31.362269 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-gbhmg" event={"ID":"b3a36399-90fc-4dbe-86f4-ffc633cd89cc","Type":"ContainerStarted","Data":"ef9f2c56728106f45f48a334b0c75bcd195d7789adcbbf2327a1d95c347c5834"} Sep 10 23:37:31.363836 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:31.363820 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zkr6f_aacd921b-5c15-4f7f-8685-cfceebfe5407/console-operator/0.log" Sep 10 23:37:31.363911 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:31.363854 2578 generic.go:356] "Generic (PLEG): container finished" podID="aacd921b-5c15-4f7f-8685-cfceebfe5407" containerID="253b18b0e9cd08db977724b91dc34cf056f6f65fc73f973dcaf237f7e6d6eec1" exitCode=255 Sep 10 23:37:31.363980 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:31.363956 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" event={"ID":"aacd921b-5c15-4f7f-8685-cfceebfe5407","Type":"ContainerDied","Data":"253b18b0e9cd08db977724b91dc34cf056f6f65fc73f973dcaf237f7e6d6eec1"} Sep 10 23:37:31.364099 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:31.364087 2578 scope.go:117] "RemoveContainer" containerID="253b18b0e9cd08db977724b91dc34cf056f6f65fc73f973dcaf237f7e6d6eec1" Sep 10 23:37:31.365309 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:31.365264 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fb4qx" event={"ID":"c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d","Type":"ContainerStarted","Data":"7fcf7020f1d660bb0a521c85b0d2c5351c766a65de67bcd05d336b8ca4b0d66f"} Sep 10 23:37:31.366521 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:31.366499 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zzfhc" event={"ID":"9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7","Type":"ContainerStarted","Data":"86cb502dd6fdb501d60a466e94978525daa838491855bc22645856ed9bc4fd9b"} Sep 10 23:37:31.377961 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:31.377907 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-gbhmg" podStartSLOduration=1.608313933 podStartE2EDuration="4.377893898s" totalImagesPullingTime="2.769579965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:31.376869239 +0000 UTC m=+66.844343578" watchObservedRunningTime="2026-09-10 23:37:31.377893898 +0000 UTC m=+66.845368223" Sep 10 23:37:32.371766 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:32.371736 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zkr6f_aacd921b-5c15-4f7f-8685-cfceebfe5407/console-operator/1.log" Sep 10 23:37:32.372220 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:32.372211 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zkr6f_aacd921b-5c15-4f7f-8685-cfceebfe5407/console-operator/0.log" Sep 10 23:37:32.372299 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:32.372246 2578 generic.go:356] "Generic (PLEG): container finished" podID="aacd921b-5c15-4f7f-8685-cfceebfe5407" containerID="8d332d987cd52a06d86277bd080bcff9d1536d3342c0462ceebe05fe446cb363" exitCode=255 Sep 10 23:37:32.372356 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:32.372329 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" event={"ID":"aacd921b-5c15-4f7f-8685-cfceebfe5407","Type":"ContainerDied","Data":"8d332d987cd52a06d86277bd080bcff9d1536d3342c0462ceebe05fe446cb363"} Sep 10 23:37:32.372407 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:32.372375 2578 scope.go:117] "RemoveContainer" containerID="253b18b0e9cd08db977724b91dc34cf056f6f65fc73f973dcaf237f7e6d6eec1" Sep 10 23:37:32.372604 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:32.372591 2578 scope.go:117] "RemoveContainer" containerID="8d332d987cd52a06d86277bd080bcff9d1536d3342c0462ceebe05fe446cb363" Sep 10 23:37:32.372796 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:32.372772 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-zkr6f_openshift-console-operator(aacd921b-5c15-4f7f-8685-cfceebfe5407)\"" pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" podUID="aacd921b-5c15-4f7f-8685-cfceebfe5407" Sep 10 23:37:32.895015 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:32.894965 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4l5c2"] Sep 10 23:37:32.898986 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:32.898560 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:37:32.901298 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:32.901274 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 23:37:32.901529 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:32.901508 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4wv72\"" Sep 10 23:37:32.901781 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:32.901762 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 23:37:32.911681 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:32.911609 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4l5c2"] Sep 10 23:37:32.956975 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:32.956919 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/79b487d6-5e92-45eb-b8fb-03abd1f439b8-data-volume\") pod \"insights-runtime-extractor-4l5c2\" (UID: \"79b487d6-5e92-45eb-b8fb-03abd1f439b8\") " pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:37:32.957162 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:32.956997 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/79b487d6-5e92-45eb-b8fb-03abd1f439b8-crio-socket\") pod \"insights-runtime-extractor-4l5c2\" (UID: \"79b487d6-5e92-45eb-b8fb-03abd1f439b8\") " pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:37:32.957162 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:32.957025 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/79b487d6-5e92-45eb-b8fb-03abd1f439b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4l5c2\" (UID: \"79b487d6-5e92-45eb-b8fb-03abd1f439b8\") " pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:37:32.957313 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:32.957168 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/79b487d6-5e92-45eb-b8fb-03abd1f439b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4l5c2\" (UID: \"79b487d6-5e92-45eb-b8fb-03abd1f439b8\") " pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:37:32.957313 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:32.957237 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ms2s\" (UniqueName: \"kubernetes.io/projected/79b487d6-5e92-45eb-b8fb-03abd1f439b8-kube-api-access-6ms2s\") pod \"insights-runtime-extractor-4l5c2\" (UID: \"79b487d6-5e92-45eb-b8fb-03abd1f439b8\") " pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:37:33.058351 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:33.058271 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/79b487d6-5e92-45eb-b8fb-03abd1f439b8-data-volume\") pod \"insights-runtime-extractor-4l5c2\" (UID: \"79b487d6-5e92-45eb-b8fb-03abd1f439b8\") " pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:37:33.058351 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:33.058328 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/79b487d6-5e92-45eb-b8fb-03abd1f439b8-crio-socket\") pod \"insights-runtime-extractor-4l5c2\" (UID: \"79b487d6-5e92-45eb-b8fb-03abd1f439b8\") " pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:37:33.058351 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:33.058355 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/79b487d6-5e92-45eb-b8fb-03abd1f439b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4l5c2\" (UID: \"79b487d6-5e92-45eb-b8fb-03abd1f439b8\") " pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:37:33.058618 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:33.058431 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/79b487d6-5e92-45eb-b8fb-03abd1f439b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4l5c2\" (UID: \"79b487d6-5e92-45eb-b8fb-03abd1f439b8\") " pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:37:33.058618 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:33.058480 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ms2s\" (UniqueName: \"kubernetes.io/projected/79b487d6-5e92-45eb-b8fb-03abd1f439b8-kube-api-access-6ms2s\") pod \"insights-runtime-extractor-4l5c2\" (UID: \"79b487d6-5e92-45eb-b8fb-03abd1f439b8\") " pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:37:33.058720 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:33.058664 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/79b487d6-5e92-45eb-b8fb-03abd1f439b8-data-volume\") pod \"insights-runtime-extractor-4l5c2\" (UID: \"79b487d6-5e92-45eb-b8fb-03abd1f439b8\") " pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:37:33.058809 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:33.058777 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 23:37:33.058809 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:33.058786 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 23:37:33.058961 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:33.058822 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/79b487d6-5e92-45eb-b8fb-03abd1f439b8-crio-socket\") pod \"insights-runtime-extractor-4l5c2\" (UID: \"79b487d6-5e92-45eb-b8fb-03abd1f439b8\") " pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:37:33.058961 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:33.058855 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79b487d6-5e92-45eb-b8fb-03abd1f439b8-kube-rbac-proxy-cm podName:79b487d6-5e92-45eb-b8fb-03abd1f439b8 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:33.558834678 +0000 UTC m=+69.026309006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/79b487d6-5e92-45eb-b8fb-03abd1f439b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4l5c2" (UID: "79b487d6-5e92-45eb-b8fb-03abd1f439b8") : configmap "kube-rbac-proxy" not found Sep 10 23:37:33.058961 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:33.058901 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b487d6-5e92-45eb-b8fb-03abd1f439b8-insights-runtime-extractor-tls podName:79b487d6-5e92-45eb-b8fb-03abd1f439b8 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:33.558888355 +0000 UTC m=+69.026362664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/79b487d6-5e92-45eb-b8fb-03abd1f439b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4l5c2" (UID: "79b487d6-5e92-45eb-b8fb-03abd1f439b8") : secret "insights-runtime-extractor-tls" not found Sep 10 23:37:33.068259 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:33.068224 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ms2s\" (UniqueName: \"kubernetes.io/projected/79b487d6-5e92-45eb-b8fb-03abd1f439b8-kube-api-access-6ms2s\") pod \"insights-runtime-extractor-4l5c2\" (UID: \"79b487d6-5e92-45eb-b8fb-03abd1f439b8\") " pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:37:33.377556 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:33.377527 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zkr6f_aacd921b-5c15-4f7f-8685-cfceebfe5407/console-operator/1.log" Sep 10 23:37:33.378040 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:33.377954 2578 scope.go:117] "RemoveContainer" containerID="8d332d987cd52a06d86277bd080bcff9d1536d3342c0462ceebe05fe446cb363" Sep 10 23:37:33.378172 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:33.378150 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-zkr6f_openshift-console-operator(aacd921b-5c15-4f7f-8685-cfceebfe5407)\"" pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" podUID="aacd921b-5c15-4f7f-8685-cfceebfe5407" Sep 10 23:37:33.563411 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:33.563373 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/79b487d6-5e92-45eb-b8fb-03abd1f439b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4l5c2\" (UID: \"79b487d6-5e92-45eb-b8fb-03abd1f439b8\") " pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:37:33.563622 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:33.563475 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/79b487d6-5e92-45eb-b8fb-03abd1f439b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4l5c2\" (UID: \"79b487d6-5e92-45eb-b8fb-03abd1f439b8\") " pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:37:33.563622 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:33.563513 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 23:37:33.563622 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:33.563598 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79b487d6-5e92-45eb-b8fb-03abd1f439b8-kube-rbac-proxy-cm podName:79b487d6-5e92-45eb-b8fb-03abd1f439b8 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:34.563576485 +0000 UTC m=+70.031050812 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/79b487d6-5e92-45eb-b8fb-03abd1f439b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4l5c2" (UID: "79b487d6-5e92-45eb-b8fb-03abd1f439b8") : configmap "kube-rbac-proxy" not found Sep 10 23:37:33.563622 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:33.563609 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 23:37:33.563842 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:33.563651 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b487d6-5e92-45eb-b8fb-03abd1f439b8-insights-runtime-extractor-tls podName:79b487d6-5e92-45eb-b8fb-03abd1f439b8 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:34.563638568 +0000 UTC m=+70.031112888 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/79b487d6-5e92-45eb-b8fb-03abd1f439b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4l5c2" (UID: "79b487d6-5e92-45eb-b8fb-03abd1f439b8") : secret "insights-runtime-extractor-tls" not found Sep 10 23:37:34.569600 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:34.569558 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/79b487d6-5e92-45eb-b8fb-03abd1f439b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4l5c2\" (UID: \"79b487d6-5e92-45eb-b8fb-03abd1f439b8\") " pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:37:34.570070 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:34.569663 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/79b487d6-5e92-45eb-b8fb-03abd1f439b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4l5c2\" (UID: \"79b487d6-5e92-45eb-b8fb-03abd1f439b8\") " pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:37:34.570070 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:34.569720 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 23:37:34.570070 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:34.569792 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b487d6-5e92-45eb-b8fb-03abd1f439b8-insights-runtime-extractor-tls podName:79b487d6-5e92-45eb-b8fb-03abd1f439b8 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:36.569773686 +0000 UTC m=+72.037247990 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/79b487d6-5e92-45eb-b8fb-03abd1f439b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4l5c2" (UID: "79b487d6-5e92-45eb-b8fb-03abd1f439b8") : secret "insights-runtime-extractor-tls" not found Sep 10 23:37:34.570070 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:34.569798 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 23:37:34.570070 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:34.569880 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79b487d6-5e92-45eb-b8fb-03abd1f439b8-kube-rbac-proxy-cm podName:79b487d6-5e92-45eb-b8fb-03abd1f439b8 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:36.569852984 +0000 UTC m=+72.037327311 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/79b487d6-5e92-45eb-b8fb-03abd1f439b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4l5c2" (UID: "79b487d6-5e92-45eb-b8fb-03abd1f439b8") : configmap "kube-rbac-proxy" not found Sep 10 23:37:34.633150 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:34.633117 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-h64sk"] Sep 10 23:37:34.636050 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:34.636030 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-h64sk" Sep 10 23:37:34.638428 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:34.638408 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 10 23:37:34.639256 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:34.639233 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 10 23:37:34.639376 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:34.639255 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 10 23:37:34.639376 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:34.639278 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 10 23:37:34.639376 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:34.639255 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9mbrh\"" Sep 10 23:37:34.643508 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:34.643486 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-h64sk"] Sep 10 23:37:34.671129 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:34.671095 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj5g9\" (UniqueName: \"kubernetes.io/projected/67a71852-1ef9-4954-a1e4-400feb696fce-kube-api-access-qj5g9\") pod \"service-ca-568869fdc5-h64sk\" (UID: \"67a71852-1ef9-4954-a1e4-400feb696fce\") " pod="openshift-service-ca/service-ca-568869fdc5-h64sk" Sep 10 23:37:34.671280 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:34.671145 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/67a71852-1ef9-4954-a1e4-400feb696fce-signing-cabundle\") pod \"service-ca-568869fdc5-h64sk\" (UID: \"67a71852-1ef9-4954-a1e4-400feb696fce\") " pod="openshift-service-ca/service-ca-568869fdc5-h64sk" Sep 10 23:37:34.671280 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:34.671164 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/67a71852-1ef9-4954-a1e4-400feb696fce-signing-key\") pod \"service-ca-568869fdc5-h64sk\" (UID: \"67a71852-1ef9-4954-a1e4-400feb696fce\") " pod="openshift-service-ca/service-ca-568869fdc5-h64sk" Sep 10 23:37:34.771872 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:34.771829 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj5g9\" (UniqueName: \"kubernetes.io/projected/67a71852-1ef9-4954-a1e4-400feb696fce-kube-api-access-qj5g9\") pod \"service-ca-568869fdc5-h64sk\" (UID: \"67a71852-1ef9-4954-a1e4-400feb696fce\") " pod="openshift-service-ca/service-ca-568869fdc5-h64sk" Sep 10 23:37:34.772075 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:34.771892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/67a71852-1ef9-4954-a1e4-400feb696fce-signing-cabundle\") pod \"service-ca-568869fdc5-h64sk\" (UID: \"67a71852-1ef9-4954-a1e4-400feb696fce\") " pod="openshift-service-ca/service-ca-568869fdc5-h64sk" Sep 10 23:37:34.772075 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:34.771919 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/67a71852-1ef9-4954-a1e4-400feb696fce-signing-key\") pod \"service-ca-568869fdc5-h64sk\" (UID: \"67a71852-1ef9-4954-a1e4-400feb696fce\") " pod="openshift-service-ca/service-ca-568869fdc5-h64sk" Sep 10 23:37:34.772623 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:34.772597 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/67a71852-1ef9-4954-a1e4-400feb696fce-signing-cabundle\") pod \"service-ca-568869fdc5-h64sk\" (UID: \"67a71852-1ef9-4954-a1e4-400feb696fce\") " pod="openshift-service-ca/service-ca-568869fdc5-h64sk" Sep 10 23:37:34.774812 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:34.774790 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/67a71852-1ef9-4954-a1e4-400feb696fce-signing-key\") pod \"service-ca-568869fdc5-h64sk\" (UID: \"67a71852-1ef9-4954-a1e4-400feb696fce\") " pod="openshift-service-ca/service-ca-568869fdc5-h64sk" Sep 10 23:37:34.779986 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:34.779967 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj5g9\" (UniqueName: \"kubernetes.io/projected/67a71852-1ef9-4954-a1e4-400feb696fce-kube-api-access-qj5g9\") pod \"service-ca-568869fdc5-h64sk\" (UID: \"67a71852-1ef9-4954-a1e4-400feb696fce\") " pod="openshift-service-ca/service-ca-568869fdc5-h64sk" Sep 10 23:37:34.947351 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:34.947317 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-h64sk" Sep 10 23:37:35.680048 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:35.680019 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dm6sw_1da41227-eab5-43fa-a97c-69f7d935cc76/dns-node-resolver/0.log" Sep 10 23:37:35.996793 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:35.996711 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-h64sk"] Sep 10 23:37:36.002614 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:37:36.002582 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a71852_1ef9_4954_a1e4_400feb696fce.slice/crio-86b3f01ecef50a51ba95e21a0edf62885ab9a1ac60f78720cc14b9d4b8dc14e5 WatchSource:0}: Error finding container 86b3f01ecef50a51ba95e21a0edf62885ab9a1ac60f78720cc14b9d4b8dc14e5: Status 404 returned error can't find the container with id 86b3f01ecef50a51ba95e21a0edf62885ab9a1ac60f78720cc14b9d4b8dc14e5 Sep 10 23:37:36.386611 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:36.386573 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fb4qx" event={"ID":"c3fb908c-f83c-4f4d-bdc1-2ad529ae5e8d","Type":"ContainerStarted","Data":"57722b2bb06ddb7c9c81094f18d18bd02acefc77079efa174cf5b7769a58663f"} Sep 10 23:37:36.386791 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:36.386711 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:37:36.387800 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:36.387778 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-h64sk" event={"ID":"67a71852-1ef9-4954-a1e4-400feb696fce","Type":"ContainerStarted","Data":"86b3f01ecef50a51ba95e21a0edf62885ab9a1ac60f78720cc14b9d4b8dc14e5"} Sep 10 23:37:36.389005 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:36.388983 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zzfhc" event={"ID":"9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7","Type":"ContainerStarted","Data":"814c0bbd1468d4f99a75e5afd40edc2add6d397fb8a5a45b50670919bfff3f01"} Sep 10 23:37:36.405920 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:36.405872 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fb4qx" podStartSLOduration=67.029733216 podStartE2EDuration="1m11.405852082s" totalImagesPullingTime="4.376118866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:36.404395403 +0000 UTC m=+71.871869728" watchObservedRunningTime="2026-09-10 23:37:36.405852082 +0000 UTC m=+71.873326406" Sep 10 23:37:36.421516 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:36.421478 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zzfhc" podStartSLOduration=66.86572741 podStartE2EDuration="1m11.42146907s" totalImagesPullingTime="4.55574166s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:36.420755445 +0000 UTC m=+71.888229769" watchObservedRunningTime="2026-09-10 23:37:36.42146907 +0000 UTC m=+71.888943395" Sep 10 23:37:36.585266 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:36.585229 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/79b487d6-5e92-45eb-b8fb-03abd1f439b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4l5c2\" (UID: \"79b487d6-5e92-45eb-b8fb-03abd1f439b8\") " pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:37:36.585414 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:36.585297 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/79b487d6-5e92-45eb-b8fb-03abd1f439b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4l5c2\" (UID: \"79b487d6-5e92-45eb-b8fb-03abd1f439b8\") " pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:37:36.585414 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:36.585368 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 23:37:36.585414 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:36.585383 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 23:37:36.585535 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:36.585423 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79b487d6-5e92-45eb-b8fb-03abd1f439b8-kube-rbac-proxy-cm podName:79b487d6-5e92-45eb-b8fb-03abd1f439b8 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:40.585409452 +0000 UTC m=+76.052883755 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/79b487d6-5e92-45eb-b8fb-03abd1f439b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4l5c2" (UID: "79b487d6-5e92-45eb-b8fb-03abd1f439b8") : configmap "kube-rbac-proxy" not found Sep 10 23:37:36.585535 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:36.585444 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b487d6-5e92-45eb-b8fb-03abd1f439b8-insights-runtime-extractor-tls podName:79b487d6-5e92-45eb-b8fb-03abd1f439b8 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:40.585429493 +0000 UTC m=+76.052903797 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/79b487d6-5e92-45eb-b8fb-03abd1f439b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4l5c2" (UID: "79b487d6-5e92-45eb-b8fb-03abd1f439b8") : secret "insights-runtime-extractor-tls" not found Sep 10 23:37:36.679563 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:36.679480 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f7fjr_7be68ac6-c59b-4afa-9974-a39f8067c1f7/node-ca/0.log" Sep 10 23:37:37.281196 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:37.281166 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-77hbh_0cb537d8-167e-4bcc-9681-1bdc99fd9cef/serve-healthcheck-canary/0.log" Sep 10 23:37:37.783817 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:37.780639 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" Sep 10 23:37:37.783817 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:37.780716 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" Sep 10 23:37:37.784244 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:37.784223 2578 scope.go:117] "RemoveContainer" containerID="8d332d987cd52a06d86277bd080bcff9d1536d3342c0462ceebe05fe446cb363" Sep 10 23:37:37.784415 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:37.784396 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-zkr6f_openshift-console-operator(aacd921b-5c15-4f7f-8685-cfceebfe5407)\"" pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" podUID="aacd921b-5c15-4f7f-8685-cfceebfe5407" Sep 10 23:37:38.396647 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:38.396595 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-h64sk" event={"ID":"67a71852-1ef9-4954-a1e4-400feb696fce","Type":"ContainerStarted","Data":"09dbc7a54e8f40456d048e4c6f32ffbff90f7f58b6df51a8b2075dd774959079"} Sep 10 23:37:38.415062 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:38.415014 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-h64sk" podStartSLOduration=2.428835837 podStartE2EDuration="4.414997197s" totalImagesPullingTime="1.98616136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:38.414215424 +0000 UTC m=+73.881689750" watchObservedRunningTime="2026-09-10 23:37:38.414997197 +0000 UTC m=+73.882471522" Sep 10 23:37:40.617047 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:40.617009 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/79b487d6-5e92-45eb-b8fb-03abd1f439b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4l5c2\" (UID: \"79b487d6-5e92-45eb-b8fb-03abd1f439b8\") " pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:37:40.617512 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:40.617104 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/79b487d6-5e92-45eb-b8fb-03abd1f439b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4l5c2\" (UID: \"79b487d6-5e92-45eb-b8fb-03abd1f439b8\") " pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:37:40.617512 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:40.617185 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 23:37:40.617512 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:40.617272 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b487d6-5e92-45eb-b8fb-03abd1f439b8-insights-runtime-extractor-tls podName:79b487d6-5e92-45eb-b8fb-03abd1f439b8 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:48.617251076 +0000 UTC m=+84.084725393 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/79b487d6-5e92-45eb-b8fb-03abd1f439b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4l5c2" (UID: "79b487d6-5e92-45eb-b8fb-03abd1f439b8") : secret "insights-runtime-extractor-tls" not found Sep 10 23:37:40.617512 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:40.617192 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 23:37:40.617512 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:40.617356 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79b487d6-5e92-45eb-b8fb-03abd1f439b8-kube-rbac-proxy-cm podName:79b487d6-5e92-45eb-b8fb-03abd1f439b8 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:48.617337239 +0000 UTC m=+84.084811562 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/79b487d6-5e92-45eb-b8fb-03abd1f439b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4l5c2" (UID: "79b487d6-5e92-45eb-b8fb-03abd1f439b8") : configmap "kube-rbac-proxy" not found Sep 10 23:37:48.688112 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:48.687996 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/79b487d6-5e92-45eb-b8fb-03abd1f439b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4l5c2\" (UID: \"79b487d6-5e92-45eb-b8fb-03abd1f439b8\") " pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:37:48.688112 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:48.688077 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/79b487d6-5e92-45eb-b8fb-03abd1f439b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4l5c2\" (UID: \"79b487d6-5e92-45eb-b8fb-03abd1f439b8\") " pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:37:48.688581 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:48.688142 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 23:37:48.688581 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:48.688216 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b487d6-5e92-45eb-b8fb-03abd1f439b8-insights-runtime-extractor-tls podName:79b487d6-5e92-45eb-b8fb-03abd1f439b8 nodeName:}" failed. No retries permitted until 2026-09-10 23:38:04.688197627 +0000 UTC m=+100.155671955 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/79b487d6-5e92-45eb-b8fb-03abd1f439b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4l5c2" (UID: "79b487d6-5e92-45eb-b8fb-03abd1f439b8") : secret "insights-runtime-extractor-tls" not found Sep 10 23:37:48.688581 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:48.688531 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/79b487d6-5e92-45eb-b8fb-03abd1f439b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4l5c2\" (UID: \"79b487d6-5e92-45eb-b8fb-03abd1f439b8\") " pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:37:49.039071 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:49.038988 2578 scope.go:117] "RemoveContainer" containerID="8d332d987cd52a06d86277bd080bcff9d1536d3342c0462ceebe05fe446cb363" Sep 10 23:37:49.435690 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:49.435658 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zkr6f_aacd921b-5c15-4f7f-8685-cfceebfe5407/console-operator/2.log" Sep 10 23:37:49.436117 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:49.436100 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zkr6f_aacd921b-5c15-4f7f-8685-cfceebfe5407/console-operator/1.log" Sep 10 23:37:49.436179 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:49.436134 2578 generic.go:356] "Generic (PLEG): container finished" podID="aacd921b-5c15-4f7f-8685-cfceebfe5407" containerID="ad7718295f117b35599f0bc986a41902e4dd6c74f9971378240131a36af62444" exitCode=255 Sep 10 23:37:49.436217 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:49.436194 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" event={"ID":"aacd921b-5c15-4f7f-8685-cfceebfe5407","Type":"ContainerDied","Data":"ad7718295f117b35599f0bc986a41902e4dd6c74f9971378240131a36af62444"} Sep 10 23:37:49.436258 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:49.436238 2578 scope.go:117] "RemoveContainer" containerID="8d332d987cd52a06d86277bd080bcff9d1536d3342c0462ceebe05fe446cb363" Sep 10 23:37:49.436559 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:49.436533 2578 scope.go:117] "RemoveContainer" containerID="ad7718295f117b35599f0bc986a41902e4dd6c74f9971378240131a36af62444" Sep 10 23:37:49.436742 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:49.436708 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-zkr6f_openshift-console-operator(aacd921b-5c15-4f7f-8685-cfceebfe5407)\"" pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" podUID="aacd921b-5c15-4f7f-8685-cfceebfe5407" Sep 10 23:37:50.440526 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:50.440493 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zkr6f_aacd921b-5c15-4f7f-8685-cfceebfe5407/console-operator/2.log" Sep 10 23:37:56.894907 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:56.894869 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-595b74c476-n5msz"] Sep 10 23:37:56.899184 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:56.899159 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-595b74c476-n5msz" Sep 10 23:37:56.900045 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:56.900022 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts"] Sep 10 23:37:56.901544 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:56.901515 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 10 23:37:56.901644 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:56.901616 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 10 23:37:56.901703 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:56.901691 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 10 23:37:56.902489 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:56.902471 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-qrb2k\"" Sep 10 23:37:56.902579 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:56.902532 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 10 23:37:56.902791 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:56.902775 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts" Sep 10 23:37:56.905015 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:56.904996 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 10 23:37:56.905109 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:56.905019 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 10 23:37:56.905277 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:56.905256 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 10 23:37:56.905353 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:56.905296 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 10 23:37:56.916161 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:56.916137 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-595b74c476-n5msz"] Sep 10 23:37:56.917168 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:56.917149 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts"] Sep 10 23:37:57.053466 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.053434 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/42f815eb-1503-42c0-b245-03f7a8802107-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-595b74c476-n5msz\" (UID: \"42f815eb-1503-42c0-b245-03f7a8802107\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-595b74c476-n5msz" Sep 10 23:37:57.053632 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.053472 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b82271ab-c316-4673-b3c8-5b9edd64bd58-hub\") pod \"cluster-proxy-proxy-agent-69df5588d6-bbnts\" (UID: \"b82271ab-c316-4673-b3c8-5b9edd64bd58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts" Sep 10 23:37:57.053632 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.053499 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsxzs\" (UniqueName: \"kubernetes.io/projected/42f815eb-1503-42c0-b245-03f7a8802107-kube-api-access-nsxzs\") pod \"managed-serviceaccount-addon-agent-595b74c476-n5msz\" (UID: \"42f815eb-1503-42c0-b245-03f7a8802107\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-595b74c476-n5msz" Sep 10 23:37:57.053632 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.053552 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b82271ab-c316-4673-b3c8-5b9edd64bd58-ca\") pod \"cluster-proxy-proxy-agent-69df5588d6-bbnts\" (UID: \"b82271ab-c316-4673-b3c8-5b9edd64bd58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts" Sep 10 23:37:57.053632 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.053593 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b82271ab-c316-4673-b3c8-5b9edd64bd58-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69df5588d6-bbnts\" (UID: \"b82271ab-c316-4673-b3c8-5b9edd64bd58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts" Sep 10 23:37:57.053632 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.053626 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b82271ab-c316-4673-b3c8-5b9edd64bd58-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69df5588d6-bbnts\" (UID: \"b82271ab-c316-4673-b3c8-5b9edd64bd58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts" Sep 10 23:37:57.053787 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.053644 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b82271ab-c316-4673-b3c8-5b9edd64bd58-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69df5588d6-bbnts\" (UID: \"b82271ab-c316-4673-b3c8-5b9edd64bd58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts" Sep 10 23:37:57.053787 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.053690 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfldp\" (UniqueName: \"kubernetes.io/projected/b82271ab-c316-4673-b3c8-5b9edd64bd58-kube-api-access-lfldp\") pod \"cluster-proxy-proxy-agent-69df5588d6-bbnts\" (UID: \"b82271ab-c316-4673-b3c8-5b9edd64bd58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts" Sep 10 23:37:57.154930 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.154844 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b82271ab-c316-4673-b3c8-5b9edd64bd58-ca\") pod \"cluster-proxy-proxy-agent-69df5588d6-bbnts\" (UID: \"b82271ab-c316-4673-b3c8-5b9edd64bd58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts" Sep 10 23:37:57.154930 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.154902 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b82271ab-c316-4673-b3c8-5b9edd64bd58-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69df5588d6-bbnts\" (UID: \"b82271ab-c316-4673-b3c8-5b9edd64bd58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts" Sep 10 23:37:57.155126 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.154954 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b82271ab-c316-4673-b3c8-5b9edd64bd58-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69df5588d6-bbnts\" (UID: \"b82271ab-c316-4673-b3c8-5b9edd64bd58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts" Sep 10 23:37:57.155126 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.155089 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b82271ab-c316-4673-b3c8-5b9edd64bd58-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69df5588d6-bbnts\" (UID: \"b82271ab-c316-4673-b3c8-5b9edd64bd58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts" Sep 10 23:37:57.155199 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.155126 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfldp\" (UniqueName: \"kubernetes.io/projected/b82271ab-c316-4673-b3c8-5b9edd64bd58-kube-api-access-lfldp\") pod \"cluster-proxy-proxy-agent-69df5588d6-bbnts\" (UID: \"b82271ab-c316-4673-b3c8-5b9edd64bd58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts" Sep 10 23:37:57.155199 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.155171 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/42f815eb-1503-42c0-b245-03f7a8802107-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-595b74c476-n5msz\" (UID: \"42f815eb-1503-42c0-b245-03f7a8802107\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-595b74c476-n5msz" Sep 10 23:37:57.155294 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.155219 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b82271ab-c316-4673-b3c8-5b9edd64bd58-hub\") pod \"cluster-proxy-proxy-agent-69df5588d6-bbnts\" (UID: \"b82271ab-c316-4673-b3c8-5b9edd64bd58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts" Sep 10 23:37:57.155659 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.155556 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsxzs\" (UniqueName: \"kubernetes.io/projected/42f815eb-1503-42c0-b245-03f7a8802107-kube-api-access-nsxzs\") pod \"managed-serviceaccount-addon-agent-595b74c476-n5msz\" (UID: \"42f815eb-1503-42c0-b245-03f7a8802107\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-595b74c476-n5msz" Sep 10 23:37:57.155877 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.155852 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b82271ab-c316-4673-b3c8-5b9edd64bd58-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69df5588d6-bbnts\" (UID: \"b82271ab-c316-4673-b3c8-5b9edd64bd58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts" Sep 10 23:37:57.157747 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.157723 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b82271ab-c316-4673-b3c8-5b9edd64bd58-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69df5588d6-bbnts\" (UID: \"b82271ab-c316-4673-b3c8-5b9edd64bd58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts" Sep 10 23:37:57.157847 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.157757 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b82271ab-c316-4673-b3c8-5b9edd64bd58-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69df5588d6-bbnts\" (UID: \"b82271ab-c316-4673-b3c8-5b9edd64bd58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts" Sep 10 23:37:57.157847 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.157767 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b82271ab-c316-4673-b3c8-5b9edd64bd58-hub\") pod \"cluster-proxy-proxy-agent-69df5588d6-bbnts\" (UID: \"b82271ab-c316-4673-b3c8-5b9edd64bd58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts" Sep 10 23:37:57.157847 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.157773 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b82271ab-c316-4673-b3c8-5b9edd64bd58-ca\") pod \"cluster-proxy-proxy-agent-69df5588d6-bbnts\" (UID: \"b82271ab-c316-4673-b3c8-5b9edd64bd58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts" Sep 10 23:37:57.158121 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.158102 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/42f815eb-1503-42c0-b245-03f7a8802107-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-595b74c476-n5msz\" (UID: \"42f815eb-1503-42c0-b245-03f7a8802107\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-595b74c476-n5msz" Sep 10 23:37:57.164479 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.164454 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfldp\" (UniqueName: \"kubernetes.io/projected/b82271ab-c316-4673-b3c8-5b9edd64bd58-kube-api-access-lfldp\") pod \"cluster-proxy-proxy-agent-69df5588d6-bbnts\" (UID: \"b82271ab-c316-4673-b3c8-5b9edd64bd58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts" Sep 10 23:37:57.164663 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.164643 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsxzs\" (UniqueName: \"kubernetes.io/projected/42f815eb-1503-42c0-b245-03f7a8802107-kube-api-access-nsxzs\") pod \"managed-serviceaccount-addon-agent-595b74c476-n5msz\" (UID: \"42f815eb-1503-42c0-b245-03f7a8802107\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-595b74c476-n5msz" Sep 10 23:37:57.218457 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.218424 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-595b74c476-n5msz" Sep 10 23:37:57.223566 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.223547 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts" Sep 10 23:37:57.499422 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.499349 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-595b74c476-n5msz"] Sep 10 23:37:57.501545 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.501519 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts"] Sep 10 23:37:57.502660 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:37:57.502639 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42f815eb_1503_42c0_b245_03f7a8802107.slice/crio-894200ffbd675c62c3334198254d2f1e8d20c15273b5f31186ff55213f7782aa WatchSource:0}: Error finding container 894200ffbd675c62c3334198254d2f1e8d20c15273b5f31186ff55213f7782aa: Status 404 returned error can't find the container with id 894200ffbd675c62c3334198254d2f1e8d20c15273b5f31186ff55213f7782aa Sep 10 23:37:57.503787 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:37:57.503765 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb82271ab_c316_4673_b3c8_5b9edd64bd58.slice/crio-e4f76ef8250319a039f79adde71759321021909482ae085806a1053ea41bb0c1 WatchSource:0}: Error finding container e4f76ef8250319a039f79adde71759321021909482ae085806a1053ea41bb0c1: Status 404 returned error can't find the container with id e4f76ef8250319a039f79adde71759321021909482ae085806a1053ea41bb0c1 Sep 10 23:37:57.780348 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.780254 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" Sep 10 23:37:57.780486 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.780353 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" Sep 10 23:37:57.780678 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:57.780664 2578 scope.go:117] "RemoveContainer" containerID="ad7718295f117b35599f0bc986a41902e4dd6c74f9971378240131a36af62444" Sep 10 23:37:57.780832 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:37:57.780813 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-zkr6f_openshift-console-operator(aacd921b-5c15-4f7f-8685-cfceebfe5407)\"" pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" podUID="aacd921b-5c15-4f7f-8685-cfceebfe5407" Sep 10 23:37:58.467769 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:58.467689 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts" event={"ID":"b82271ab-c316-4673-b3c8-5b9edd64bd58","Type":"ContainerStarted","Data":"e4f76ef8250319a039f79adde71759321021909482ae085806a1053ea41bb0c1"} Sep 10 23:37:58.471365 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:37:58.471297 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-595b74c476-n5msz" event={"ID":"42f815eb-1503-42c0-b245-03f7a8802107","Type":"ContainerStarted","Data":"894200ffbd675c62c3334198254d2f1e8d20c15273b5f31186ff55213f7782aa"} Sep 10 23:38:01.484002 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:01.483958 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts" event={"ID":"b82271ab-c316-4673-b3c8-5b9edd64bd58","Type":"ContainerStarted","Data":"b3d2d3e3cade2935a8bead2ec8bbbd3bee36c486fcbc0be9fd79e6f52a321da1"} Sep 10 23:38:01.484002 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:01.484007 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts" event={"ID":"b82271ab-c316-4673-b3c8-5b9edd64bd58","Type":"ContainerStarted","Data":"45b1b4ed2581e530dd8660ae67c28e73c3ef6808e8bba364809f46e1db27295b"} Sep 10 23:38:01.484481 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:01.484025 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts" event={"ID":"b82271ab-c316-4673-b3c8-5b9edd64bd58","Type":"ContainerStarted","Data":"d87b03890b1f1a4ab11cf3249b2de1ef42f1f3568e2218c0941d9aebf84b6d5f"} Sep 10 23:38:01.485361 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:01.485338 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-595b74c476-n5msz" event={"ID":"42f815eb-1503-42c0-b245-03f7a8802107","Type":"ContainerStarted","Data":"0cd0b94127b71cf90f1f7d7a43a294fbbc08f03db7c28f0dfd6aa9bf51d8c9af"} Sep 10 23:38:01.502073 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:01.502020 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69df5588d6-bbnts" podStartSLOduration=1.917892638 podStartE2EDuration="5.502005028s" totalImagesPullingTime="3.58411239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:38:01.50081933 +0000 UTC m=+96.968293655" watchObservedRunningTime="2026-09-10 23:38:01.502005028 +0000 UTC m=+96.969479680" Sep 10 23:38:01.514141 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:01.514100 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-595b74c476-n5msz" podStartSLOduration=1.942989409 podStartE2EDuration="5.51408676s" totalImagesPullingTime="3.571097351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:38:01.513037706 +0000 UTC m=+96.980512031" watchObservedRunningTime="2026-09-10 23:38:01.51408676 +0000 UTC m=+96.981561085" Sep 10 23:38:03.311471 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:03.311434 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-registry-tls\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:38:03.311471 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:03.311484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b74b357-db9d-4962-9eda-ff644ef6f739-metrics-tls\") pod \"dns-default-tmscj\" (UID: \"6b74b357-db9d-4962-9eda-ff644ef6f739\") " pod="openshift-dns/dns-default-tmscj" Sep 10 23:38:03.313799 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:03.313776 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b74b357-db9d-4962-9eda-ff644ef6f739-metrics-tls\") pod \"dns-default-tmscj\" (UID: \"6b74b357-db9d-4962-9eda-ff644ef6f739\") " pod="openshift-dns/dns-default-tmscj" Sep 10 23:38:03.313870 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:03.313775 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-registry-tls\") pod \"image-registry-677fbc4d-5cv5g\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:38:03.577623 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:03.577542 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-67vss\"" Sep 10 23:38:03.585759 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:03.585741 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:38:03.596821 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:03.596795 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6vpbd\"" Sep 10 23:38:03.604693 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:03.604668 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tmscj" Sep 10 23:38:03.890662 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:03.890632 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-677fbc4d-5cv5g"] Sep 10 23:38:03.893781 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:38:03.893752 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19629720_a2c6_4554_abe2_cc664a296910.slice/crio-13c6e7baee229f4e5e552daa65592deaf615ab6cb4bfe4ea995709a2fd26f134 WatchSource:0}: Error finding container 13c6e7baee229f4e5e552daa65592deaf615ab6cb4bfe4ea995709a2fd26f134: Status 404 returned error can't find the container with id 13c6e7baee229f4e5e552daa65592deaf615ab6cb4bfe4ea995709a2fd26f134 Sep 10 23:38:03.909021 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:03.908922 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tmscj"] Sep 10 23:38:03.911175 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:38:03.911147 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b74b357_db9d_4962_9eda_ff644ef6f739.slice/crio-93e68eb684f02f05d47a040a22f283b0db9ced276da8769eca7c05ce3e9a2b32 WatchSource:0}: Error finding container 93e68eb684f02f05d47a040a22f283b0db9ced276da8769eca7c05ce3e9a2b32: Status 404 returned error can't find the container with id 93e68eb684f02f05d47a040a22f283b0db9ced276da8769eca7c05ce3e9a2b32 Sep 10 23:38:04.495887 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:04.495848 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tmscj" event={"ID":"6b74b357-db9d-4962-9eda-ff644ef6f739","Type":"ContainerStarted","Data":"93e68eb684f02f05d47a040a22f283b0db9ced276da8769eca7c05ce3e9a2b32"} Sep 10 23:38:04.497541 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:04.497510 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" event={"ID":"19629720-a2c6-4554-abe2-cc664a296910","Type":"ContainerStarted","Data":"d0f91406e5e25529c503fd1528aee3bd74a5dd67d86a740e54ad0e0d4dc4d650"} Sep 10 23:38:04.497683 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:04.497554 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" event={"ID":"19629720-a2c6-4554-abe2-cc664a296910","Type":"ContainerStarted","Data":"13c6e7baee229f4e5e552daa65592deaf615ab6cb4bfe4ea995709a2fd26f134"} Sep 10 23:38:04.497683 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:04.497584 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:38:04.518770 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:04.518710 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" podStartSLOduration=99.518691207 podStartE2EDuration="1m39.518691207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:38:04.517080957 +0000 UTC m=+99.984555284" watchObservedRunningTime="2026-09-10 23:38:04.518691207 +0000 UTC m=+99.986165532" Sep 10 23:38:04.726637 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:04.726608 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/79b487d6-5e92-45eb-b8fb-03abd1f439b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4l5c2\" (UID: \"79b487d6-5e92-45eb-b8fb-03abd1f439b8\") " pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:38:04.729341 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:04.729313 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/79b487d6-5e92-45eb-b8fb-03abd1f439b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4l5c2\" (UID: \"79b487d6-5e92-45eb-b8fb-03abd1f439b8\") " pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:38:05.010970 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:05.010917 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4l5c2" Sep 10 23:38:05.296171 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:05.296090 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4l5c2"] Sep 10 23:38:05.602655 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:38:05.602587 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b487d6_5e92_45eb_b8fb_03abd1f439b8.slice/crio-0f9010219edfe45848b9440d8890dc831b7c86c6beb2ce014bb2064210ebdf47 WatchSource:0}: Error finding container 0f9010219edfe45848b9440d8890dc831b7c86c6beb2ce014bb2064210ebdf47: Status 404 returned error can't find the container with id 0f9010219edfe45848b9440d8890dc831b7c86c6beb2ce014bb2064210ebdf47 Sep 10 23:38:06.507357 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:06.507278 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4l5c2" event={"ID":"79b487d6-5e92-45eb-b8fb-03abd1f439b8","Type":"ContainerStarted","Data":"5cf2814a4f1d090fc944dcd4ca17eba8e941339815fe95c491257c4eb87bc730"} Sep 10 23:38:06.507505 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:06.507377 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4l5c2" event={"ID":"79b487d6-5e92-45eb-b8fb-03abd1f439b8","Type":"ContainerStarted","Data":"0f9010219edfe45848b9440d8890dc831b7c86c6beb2ce014bb2064210ebdf47"} Sep 10 23:38:06.508919 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:06.508896 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tmscj" event={"ID":"6b74b357-db9d-4962-9eda-ff644ef6f739","Type":"ContainerStarted","Data":"39f48f3d8405035f3a6a572fde1c86cb1fd3b48730cfd4b11d755ecf1778fdc4"} Sep 10 23:38:06.509051 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:06.508958 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tmscj" event={"ID":"6b74b357-db9d-4962-9eda-ff644ef6f739","Type":"ContainerStarted","Data":"2628f0da1b1c4010dd2158464c00694dc6b31f115545c4c4c0f41e9f31542f64"} Sep 10 23:38:06.509094 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:06.509058 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tmscj" Sep 10 23:38:06.528057 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:06.528010 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tmscj" podStartSLOduration=65.470382828 podStartE2EDuration="1m7.527993684s" totalImagesPullingTime="2.057610856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:38:06.527177653 +0000 UTC m=+101.994651989" watchObservedRunningTime="2026-09-10 23:38:06.527993684 +0000 UTC m=+101.995468025" Sep 10 23:38:07.394468 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:07.394427 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fb4qx" Sep 10 23:38:07.515682 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:07.515644 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4l5c2" event={"ID":"79b487d6-5e92-45eb-b8fb-03abd1f439b8","Type":"ContainerStarted","Data":"48cc550456d201965fd09481830df9a7271e463ee504b722b75754915c5d88d3"} Sep 10 23:38:08.521196 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:08.521168 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4l5c2" event={"ID":"79b487d6-5e92-45eb-b8fb-03abd1f439b8","Type":"ContainerStarted","Data":"79880c752a4a05672d4bff51a4f84bfb94ae7c5c2241b93a247e74d253b089e0"} Sep 10 23:38:09.542457 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:09.542408 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4l5c2" podStartSLOduration=34.962683775 podStartE2EDuration="37.542390282s" totalImagesPullingTime="2.579706507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:37:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:38:09.540589375 +0000 UTC m=+105.008063701" watchObservedRunningTime="2026-09-10 23:38:09.542390282 +0000 UTC m=+105.009864606" Sep 10 23:38:11.572392 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.572350 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-crtrr"] Sep 10 23:38:11.575678 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.575657 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:11.578179 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.578154 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 23:38:11.578300 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.578154 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 23:38:11.578300 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.578211 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 23:38:11.578300 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.578239 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 23:38:11.578508 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.578495 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 23:38:11.579047 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.579028 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 23:38:11.579143 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.579097 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v8rxg\"" Sep 10 23:38:11.675505 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.675471 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-node-exporter-tls\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:11.675690 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.675515 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-node-exporter-wtmp\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:11.675690 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.675546 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl5z2\" (UniqueName: \"kubernetes.io/projected/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-kube-api-access-vl5z2\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:11.675690 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.675571 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-root\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:11.675690 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.675597 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-node-exporter-accelerators-collector-config\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:11.675690 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.675666 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-node-exporter-textfile\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:11.675924 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.675709 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-metrics-client-ca\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:11.675924 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.675742 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-sys\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:11.675924 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.675819 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:11.776605 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.776573 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-metrics-client-ca\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:11.776770 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.776618 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-sys\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:11.776770 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.776696 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-sys\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:11.776770 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.776757 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:11.776919 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.776842 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-node-exporter-tls\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:11.776919 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.776871 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-node-exporter-wtmp\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:11.776919 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.776899 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl5z2\" (UniqueName: \"kubernetes.io/projected/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-kube-api-access-vl5z2\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:11.777092 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.776927 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-root\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:11.777092 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.776966 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-node-exporter-accelerators-collector-config\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:11.777092 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.777008 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-node-exporter-textfile\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:11.777689 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.777246 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-metrics-client-ca\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:11.777689 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.777294 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-node-exporter-textfile\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:11.777689 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.777355 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-root\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:11.777689 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:38:11.777358 2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 10 23:38:11.777689 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.777381 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-node-exporter-wtmp\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:11.777689 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:38:11.777446 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-node-exporter-tls podName:4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72 nodeName:}" failed. No retries permitted until 2026-09-10 23:38:12.277425666 +0000 UTC m=+107.744899969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-node-exporter-tls") pod "node-exporter-crtrr" (UID: "4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72") : secret "node-exporter-tls" not found Sep 10 23:38:11.778093 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.777779 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-node-exporter-accelerators-collector-config\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:11.779290 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.779258 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:11.793778 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:11.793751 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl5z2\" (UniqueName: \"kubernetes.io/projected/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-kube-api-access-vl5z2\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:12.038977 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:12.038930 2578 scope.go:117] "RemoveContainer" containerID="ad7718295f117b35599f0bc986a41902e4dd6c74f9971378240131a36af62444" Sep 10 23:38:12.281778 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:12.281740 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-node-exporter-tls\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:12.283977 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:12.283952 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72-node-exporter-tls\") pod \"node-exporter-crtrr\" (UID: \"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72\") " pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:12.485367 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:12.485333 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-crtrr" Sep 10 23:38:12.494192 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:38:12.494164 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b5a2d4f_1293_4bb0_bdbb_a67a3dca3e72.slice/crio-c926f009a6ae2b11af87d26f584c126646bf8a085f54af82052690a7c7a09baf WatchSource:0}: Error finding container c926f009a6ae2b11af87d26f584c126646bf8a085f54af82052690a7c7a09baf: Status 404 returned error can't find the container with id c926f009a6ae2b11af87d26f584c126646bf8a085f54af82052690a7c7a09baf Sep 10 23:38:12.542737 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:12.542706 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zkr6f_aacd921b-5c15-4f7f-8685-cfceebfe5407/console-operator/2.log" Sep 10 23:38:12.542872 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:12.542827 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" event={"ID":"aacd921b-5c15-4f7f-8685-cfceebfe5407","Type":"ContainerStarted","Data":"7242b8c07fe10dce40e91ede97f2f9916dce6d49791781022b81631c841f6815"} Sep 10 23:38:12.543219 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:12.543150 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" Sep 10 23:38:12.544239 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:12.544215 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-crtrr" event={"ID":"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72","Type":"ContainerStarted","Data":"c926f009a6ae2b11af87d26f584c126646bf8a085f54af82052690a7c7a09baf"} Sep 10 23:38:12.560779 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:12.560730 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" podStartSLOduration=42.705822836 podStartE2EDuration="45.560712795s" totalImagesPullingTime="2.854889959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:38:12.560513998 +0000 UTC m=+108.027988335" watchObservedRunningTime="2026-09-10 23:38:12.560712795 +0000 UTC m=+108.028187101" Sep 10 23:38:12.628320 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:12.628070 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-zkr6f" Sep 10 23:38:13.549816 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:13.549783 2578 generic.go:356] "Generic (PLEG): container finished" podID="4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72" containerID="0702c6aa439e7cf94caeabe63948cdfffa3ba367259f61a7315ac8aeed136f63" exitCode=0 Sep 10 23:38:13.550004 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:13.549873 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-crtrr" event={"ID":"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72","Type":"ContainerDied","Data":"0702c6aa439e7cf94caeabe63948cdfffa3ba367259f61a7315ac8aeed136f63"} Sep 10 23:38:14.555988 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:14.555951 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-crtrr" event={"ID":"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72","Type":"ContainerStarted","Data":"47cec4d8ec7de5fb46c5aeba3520e1a09795110d177a31b358907289baddffaf"} Sep 10 23:38:14.556390 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:14.556011 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-crtrr" event={"ID":"4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72","Type":"ContainerStarted","Data":"71c0d9a35aeff7e04cd692eb3d765d6efff54fd1946e29b9094311c3bf82d8d1"} Sep 10 23:38:16.518533 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:16.518501 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tmscj" Sep 10 23:38:16.541519 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:16.541465 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-crtrr" podStartSLOduration=4.630739899 podStartE2EDuration="5.541445806s" totalImagesPullingTime="848.528191ms" totalInitContainerRuntime="62.177716ms" isStatefulPod=true podCreationTimestamp="2026-09-10 23:38:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:38:14.574078431 +0000 UTC m=+110.041552756" watchObservedRunningTime="2026-09-10 23:38:16.541445806 +0000 UTC m=+112.008920134" Sep 10 23:38:18.946801 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:18.946765 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-677fbc4d-5cv5g"] Sep 10 23:38:28.951919 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:28.951890 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:38:34.841165 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:34.841132 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a06f8008-28ac-4cb9-8310-f5c42b169d10-metrics-certs\") pod \"network-metrics-daemon-fmvxw\" (UID: \"a06f8008-28ac-4cb9-8310-f5c42b169d10\") " pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:38:34.843427 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:34.843407 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a06f8008-28ac-4cb9-8310-f5c42b169d10-metrics-certs\") pod \"network-metrics-daemon-fmvxw\" (UID: \"a06f8008-28ac-4cb9-8310-f5c42b169d10\") " pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:38:34.965885 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:34.965856 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cql7l\"" Sep 10 23:38:34.974263 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:34.974242 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fmvxw" Sep 10 23:38:35.242268 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:35.242246 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fmvxw"] Sep 10 23:38:35.244655 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:38:35.244617 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06f8008_28ac_4cb9_8310_f5c42b169d10.slice/crio-aef54b02e9793eb7414ff7893855a6340bea0cb61865f12c5ef6ff6e576360d9 WatchSource:0}: Error finding container aef54b02e9793eb7414ff7893855a6340bea0cb61865f12c5ef6ff6e576360d9: Status 404 returned error can't find the container with id aef54b02e9793eb7414ff7893855a6340bea0cb61865f12c5ef6ff6e576360d9 Sep 10 23:38:35.632730 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:35.632694 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fmvxw" event={"ID":"a06f8008-28ac-4cb9-8310-f5c42b169d10","Type":"ContainerStarted","Data":"aef54b02e9793eb7414ff7893855a6340bea0cb61865f12c5ef6ff6e576360d9"} Sep 10 23:38:36.639902 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:36.639867 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fmvxw" event={"ID":"a06f8008-28ac-4cb9-8310-f5c42b169d10","Type":"ContainerStarted","Data":"5e13984d04db3d9bd23d5e1d6a6bb0a8a6e64aad9348416ef5b595d9c8eadd80"} Sep 10 23:38:36.640274 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:36.639923 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fmvxw" event={"ID":"a06f8008-28ac-4cb9-8310-f5c42b169d10","Type":"ContainerStarted","Data":"691e6f9c2f0f78021796f61521785c1c85cb90909745e8520d4d8dace1766cab"} Sep 10 23:38:36.655754 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:36.655705 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fmvxw" podStartSLOduration=130.665541226 podStartE2EDuration="2m11.655691126s" totalImagesPullingTime="990.1499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:38:36.655113602 +0000 UTC m=+132.122587927" watchObservedRunningTime="2026-09-10 23:38:36.655691126 +0000 UTC m=+132.123165451" Sep 10 23:38:43.965823 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:43.965762 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" podUID="19629720-a2c6-4554-abe2-cc664a296910" containerName="registry" containerID="cri-o://d0f91406e5e25529c503fd1528aee3bd74a5dd67d86a740e54ad0e0d4dc4d650" gracePeriod=30 Sep 10 23:38:44.204520 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.204496 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:38:44.307993 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.307911 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/19629720-a2c6-4554-abe2-cc664a296910-registry-certificates\") pod \"19629720-a2c6-4554-abe2-cc664a296910\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " Sep 10 23:38:44.307993 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.307965 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/19629720-a2c6-4554-abe2-cc664a296910-installation-pull-secrets\") pod \"19629720-a2c6-4554-abe2-cc664a296910\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " Sep 10 23:38:44.308199 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.307997 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/19629720-a2c6-4554-abe2-cc664a296910-ca-trust-extracted\") pod \"19629720-a2c6-4554-abe2-cc664a296910\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " Sep 10 23:38:44.308199 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.308028 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtgwm\" (UniqueName: \"kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-kube-api-access-mtgwm\") pod \"19629720-a2c6-4554-abe2-cc664a296910\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " Sep 10 23:38:44.308199 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.308056 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/19629720-a2c6-4554-abe2-cc664a296910-image-registry-private-configuration\") pod \"19629720-a2c6-4554-abe2-cc664a296910\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " Sep 10 23:38:44.308199 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.308144 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-bound-sa-token\") pod \"19629720-a2c6-4554-abe2-cc664a296910\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " Sep 10 23:38:44.308396 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.308222 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-registry-tls\") pod \"19629720-a2c6-4554-abe2-cc664a296910\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " Sep 10 23:38:44.308396 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.308256 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19629720-a2c6-4554-abe2-cc664a296910-trusted-ca\") pod \"19629720-a2c6-4554-abe2-cc664a296910\" (UID: \"19629720-a2c6-4554-abe2-cc664a296910\") " Sep 10 23:38:44.308396 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.308330 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19629720-a2c6-4554-abe2-cc664a296910-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "19629720-a2c6-4554-abe2-cc664a296910" (UID: "19629720-a2c6-4554-abe2-cc664a296910"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:38:44.308551 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.308489 2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/19629720-a2c6-4554-abe2-cc664a296910-registry-certificates\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:38:44.308740 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.308708 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19629720-a2c6-4554-abe2-cc664a296910-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "19629720-a2c6-4554-abe2-cc664a296910" (UID: "19629720-a2c6-4554-abe2-cc664a296910"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:38:44.310622 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.310536 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "19629720-a2c6-4554-abe2-cc664a296910" (UID: "19629720-a2c6-4554-abe2-cc664a296910"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:38:44.310742 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.310679 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "19629720-a2c6-4554-abe2-cc664a296910" (UID: "19629720-a2c6-4554-abe2-cc664a296910"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:38:44.310818 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.310780 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19629720-a2c6-4554-abe2-cc664a296910-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "19629720-a2c6-4554-abe2-cc664a296910" (UID: "19629720-a2c6-4554-abe2-cc664a296910"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 23:38:44.310873 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.310831 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-kube-api-access-mtgwm" (OuterVolumeSpecName: "kube-api-access-mtgwm") pod "19629720-a2c6-4554-abe2-cc664a296910" (UID: "19629720-a2c6-4554-abe2-cc664a296910"). InnerVolumeSpecName "kube-api-access-mtgwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:38:44.310873 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.310848 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19629720-a2c6-4554-abe2-cc664a296910-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "19629720-a2c6-4554-abe2-cc664a296910" (UID: "19629720-a2c6-4554-abe2-cc664a296910"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 23:38:44.319556 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.319528 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19629720-a2c6-4554-abe2-cc664a296910-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "19629720-a2c6-4554-abe2-cc664a296910" (UID: "19629720-a2c6-4554-abe2-cc664a296910"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:38:44.409372 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.409343 2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/19629720-a2c6-4554-abe2-cc664a296910-installation-pull-secrets\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:38:44.409372 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.409368 2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/19629720-a2c6-4554-abe2-cc664a296910-ca-trust-extracted\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:38:44.409372 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.409377 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtgwm\" (UniqueName: \"kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-kube-api-access-mtgwm\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:38:44.409569 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.409386 2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/19629720-a2c6-4554-abe2-cc664a296910-image-registry-private-configuration\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:38:44.409569 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.409395 2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-bound-sa-token\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:38:44.409569 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.409403 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19629720-a2c6-4554-abe2-cc664a296910-registry-tls\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:38:44.409569 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.409410 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19629720-a2c6-4554-abe2-cc664a296910-trusted-ca\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:38:44.666909 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.666869 2578 generic.go:356] "Generic (PLEG): container finished" podID="19629720-a2c6-4554-abe2-cc664a296910" containerID="d0f91406e5e25529c503fd1528aee3bd74a5dd67d86a740e54ad0e0d4dc4d650" exitCode=0 Sep 10 23:38:44.667060 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.666916 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" Sep 10 23:38:44.667060 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.666917 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" event={"ID":"19629720-a2c6-4554-abe2-cc664a296910","Type":"ContainerDied","Data":"d0f91406e5e25529c503fd1528aee3bd74a5dd67d86a740e54ad0e0d4dc4d650"} Sep 10 23:38:44.667060 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.667047 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-677fbc4d-5cv5g" event={"ID":"19629720-a2c6-4554-abe2-cc664a296910","Type":"ContainerDied","Data":"13c6e7baee229f4e5e552daa65592deaf615ab6cb4bfe4ea995709a2fd26f134"} Sep 10 23:38:44.667176 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.667070 2578 scope.go:117] "RemoveContainer" containerID="d0f91406e5e25529c503fd1528aee3bd74a5dd67d86a740e54ad0e0d4dc4d650" Sep 10 23:38:44.675159 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.675143 2578 scope.go:117] "RemoveContainer" containerID="d0f91406e5e25529c503fd1528aee3bd74a5dd67d86a740e54ad0e0d4dc4d650" Sep 10 23:38:44.675426 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:38:44.675408 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0f91406e5e25529c503fd1528aee3bd74a5dd67d86a740e54ad0e0d4dc4d650\": container with ID starting with d0f91406e5e25529c503fd1528aee3bd74a5dd67d86a740e54ad0e0d4dc4d650 not found: ID does not exist" containerID="d0f91406e5e25529c503fd1528aee3bd74a5dd67d86a740e54ad0e0d4dc4d650" Sep 10 23:38:44.675466 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.675437 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f91406e5e25529c503fd1528aee3bd74a5dd67d86a740e54ad0e0d4dc4d650"} err="failed to get container status \"d0f91406e5e25529c503fd1528aee3bd74a5dd67d86a740e54ad0e0d4dc4d650\": rpc error: code = NotFound desc = could not find container \"d0f91406e5e25529c503fd1528aee3bd74a5dd67d86a740e54ad0e0d4dc4d650\": container with ID starting with d0f91406e5e25529c503fd1528aee3bd74a5dd67d86a740e54ad0e0d4dc4d650 not found: ID does not exist" Sep 10 23:38:44.686294 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.686272 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-677fbc4d-5cv5g"] Sep 10 23:38:44.689840 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:44.689819 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-677fbc4d-5cv5g"] Sep 10 23:38:45.044760 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:45.044684 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19629720-a2c6-4554-abe2-cc664a296910" path="/var/lib/kubelet/pods/19629720-a2c6-4554-abe2-cc664a296910/volumes" Sep 10 23:38:51.698618 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:51.698589 2578 generic.go:356] "Generic (PLEG): container finished" podID="b3a36399-90fc-4dbe-86f4-ffc633cd89cc" containerID="ef9f2c56728106f45f48a334b0c75bcd195d7789adcbbf2327a1d95c347c5834" exitCode=0 Sep 10 23:38:51.698974 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:51.698649 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-gbhmg" event={"ID":"b3a36399-90fc-4dbe-86f4-ffc633cd89cc","Type":"ContainerDied","Data":"ef9f2c56728106f45f48a334b0c75bcd195d7789adcbbf2327a1d95c347c5834"} Sep 10 23:38:51.698974 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:51.698966 2578 scope.go:117] "RemoveContainer" containerID="ef9f2c56728106f45f48a334b0c75bcd195d7789adcbbf2327a1d95c347c5834" Sep 10 23:38:52.203702 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:52.203671 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-77hbh_0cb537d8-167e-4bcc-9681-1bdc99fd9cef/serve-healthcheck-canary/0.log" Sep 10 23:38:52.704358 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:38:52.704322 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-gbhmg" event={"ID":"b3a36399-90fc-4dbe-86f4-ffc633cd89cc","Type":"ContainerStarted","Data":"0c83a564a9c8c02f65e81be0bb649f9108987e3e7b5b56a792ac67bb647e242d"} Sep 10 23:41:05.441266 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:05.441232 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xk65f"] Sep 10 23:41:05.441656 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:05.441471 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19629720-a2c6-4554-abe2-cc664a296910" containerName="registry" Sep 10 23:41:05.441656 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:05.441482 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="19629720-a2c6-4554-abe2-cc664a296910" containerName="registry" Sep 10 23:41:05.441656 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:05.441528 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="19629720-a2c6-4554-abe2-cc664a296910" containerName="registry" Sep 10 23:41:05.444130 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:05.444111 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xk65f" Sep 10 23:41:05.446607 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:05.446580 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 10 23:41:05.446715 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:05.446613 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 10 23:41:05.446715 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:05.446685 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-t5fxf\"" Sep 10 23:41:05.457406 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:05.457386 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xk65f"] Sep 10 23:41:05.583127 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:05.583091 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxqfn\" (UniqueName: \"kubernetes.io/projected/f5d0149d-ec70-42e8-ada8-5fc1e707c365-kube-api-access-rxqfn\") pod \"cert-manager-operator-controller-manager-55c444bf86-xk65f\" (UID: \"f5d0149d-ec70-42e8-ada8-5fc1e707c365\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xk65f" Sep 10 23:41:05.583127 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:05.583127 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5d0149d-ec70-42e8-ada8-5fc1e707c365-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-xk65f\" (UID: \"f5d0149d-ec70-42e8-ada8-5fc1e707c365\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xk65f" Sep 10 23:41:05.683444 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:05.683420 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxqfn\" (UniqueName: \"kubernetes.io/projected/f5d0149d-ec70-42e8-ada8-5fc1e707c365-kube-api-access-rxqfn\") pod \"cert-manager-operator-controller-manager-55c444bf86-xk65f\" (UID: \"f5d0149d-ec70-42e8-ada8-5fc1e707c365\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xk65f" Sep 10 23:41:05.683547 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:05.683454 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5d0149d-ec70-42e8-ada8-5fc1e707c365-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-xk65f\" (UID: \"f5d0149d-ec70-42e8-ada8-5fc1e707c365\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xk65f" Sep 10 23:41:05.683774 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:05.683759 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5d0149d-ec70-42e8-ada8-5fc1e707c365-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-xk65f\" (UID: \"f5d0149d-ec70-42e8-ada8-5fc1e707c365\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xk65f" Sep 10 23:41:05.694831 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:05.694784 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxqfn\" (UniqueName: \"kubernetes.io/projected/f5d0149d-ec70-42e8-ada8-5fc1e707c365-kube-api-access-rxqfn\") pod \"cert-manager-operator-controller-manager-55c444bf86-xk65f\" (UID: \"f5d0149d-ec70-42e8-ada8-5fc1e707c365\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xk65f" Sep 10 23:41:05.752608 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:05.752579 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xk65f" Sep 10 23:41:06.028704 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:06.028673 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xk65f"] Sep 10 23:41:06.030200 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:41:06.030169 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d0149d_ec70_42e8_ada8_5fc1e707c365.slice/crio-b9e9f717119a512d18bf3d5abb53e15de623e2a330ce698e82bb522cfdcf2dba WatchSource:0}: Error finding container b9e9f717119a512d18bf3d5abb53e15de623e2a330ce698e82bb522cfdcf2dba: Status 404 returned error can't find the container with id b9e9f717119a512d18bf3d5abb53e15de623e2a330ce698e82bb522cfdcf2dba Sep 10 23:41:06.137096 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:06.137067 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xk65f" event={"ID":"f5d0149d-ec70-42e8-ada8-5fc1e707c365","Type":"ContainerStarted","Data":"b9e9f717119a512d18bf3d5abb53e15de623e2a330ce698e82bb522cfdcf2dba"} Sep 10 23:41:09.151595 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:09.151563 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xk65f" event={"ID":"f5d0149d-ec70-42e8-ada8-5fc1e707c365","Type":"ContainerStarted","Data":"7477008abb072fca78714fe1bb57ca2a3d9def2ef4e2525ac7a3e2ecc91ea1bb"} Sep 10 23:41:09.173789 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:09.173696 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xk65f" podStartSLOduration=1.500665606 podStartE2EDuration="4.173680566s" totalImagesPullingTime="2.67301496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:41:09.172574134 +0000 UTC m=+284.640048475" watchObservedRunningTime="2026-09-10 23:41:09.173680566 +0000 UTC m=+284.641154891" Sep 10 23:41:17.421753 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:17.421718 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-sxtjp"] Sep 10 23:41:17.424711 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:17.424690 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-sxtjp" Sep 10 23:41:17.427000 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:17.426978 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 10 23:41:17.427089 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:17.426978 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-fkgrn\"" Sep 10 23:41:17.427089 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:17.427055 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 10 23:41:17.435255 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:17.435234 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-sxtjp"] Sep 10 23:41:17.569434 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:17.569395 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cdc2317d-f3f8-4a0a-8441-86cb0c326d04-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-sxtjp\" (UID: \"cdc2317d-f3f8-4a0a-8441-86cb0c326d04\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sxtjp" Sep 10 23:41:17.569598 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:17.569454 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j92c6\" (UniqueName: \"kubernetes.io/projected/cdc2317d-f3f8-4a0a-8441-86cb0c326d04-kube-api-access-j92c6\") pod \"cert-manager-webhook-6486ffd997-sxtjp\" (UID: \"cdc2317d-f3f8-4a0a-8441-86cb0c326d04\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sxtjp" Sep 10 23:41:17.670378 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:17.670344 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cdc2317d-f3f8-4a0a-8441-86cb0c326d04-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-sxtjp\" (UID: \"cdc2317d-f3f8-4a0a-8441-86cb0c326d04\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sxtjp" Sep 10 23:41:17.670538 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:17.670394 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j92c6\" (UniqueName: \"kubernetes.io/projected/cdc2317d-f3f8-4a0a-8441-86cb0c326d04-kube-api-access-j92c6\") pod \"cert-manager-webhook-6486ffd997-sxtjp\" (UID: \"cdc2317d-f3f8-4a0a-8441-86cb0c326d04\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sxtjp" Sep 10 23:41:17.678472 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:17.678411 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cdc2317d-f3f8-4a0a-8441-86cb0c326d04-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-sxtjp\" (UID: \"cdc2317d-f3f8-4a0a-8441-86cb0c326d04\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sxtjp" Sep 10 23:41:17.678472 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:17.678447 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j92c6\" (UniqueName: \"kubernetes.io/projected/cdc2317d-f3f8-4a0a-8441-86cb0c326d04-kube-api-access-j92c6\") pod \"cert-manager-webhook-6486ffd997-sxtjp\" (UID: \"cdc2317d-f3f8-4a0a-8441-86cb0c326d04\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sxtjp" Sep 10 23:41:17.733268 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:17.733240 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-sxtjp" Sep 10 23:41:18.001327 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:18.001252 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-sxtjp"] Sep 10 23:41:18.004512 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:41:18.004473 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdc2317d_f3f8_4a0a_8441_86cb0c326d04.slice/crio-93d06a628033decb6b798114279f9b95fbfb5fd322ba47a70555eb8113bbd57f WatchSource:0}: Error finding container 93d06a628033decb6b798114279f9b95fbfb5fd322ba47a70555eb8113bbd57f: Status 404 returned error can't find the container with id 93d06a628033decb6b798114279f9b95fbfb5fd322ba47a70555eb8113bbd57f Sep 10 23:41:18.185609 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:18.185569 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-sxtjp" event={"ID":"cdc2317d-f3f8-4a0a-8441-86cb0c326d04","Type":"ContainerStarted","Data":"93d06a628033decb6b798114279f9b95fbfb5fd322ba47a70555eb8113bbd57f"} Sep 10 23:41:21.199299 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:21.199260 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-sxtjp" event={"ID":"cdc2317d-f3f8-4a0a-8441-86cb0c326d04","Type":"ContainerStarted","Data":"0605ec589ca3a0182ae5f7d8f206f1d8250aa3f1e615148e01ce78ba7cf17588"} Sep 10 23:41:21.199691 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:21.199388 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-sxtjp" Sep 10 23:41:21.216170 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:21.216128 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-sxtjp" podStartSLOduration=1.57103813 podStartE2EDuration="4.216113888s" totalImagesPullingTime="2.645075758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:41:21.215204898 +0000 UTC m=+296.682679225" watchObservedRunningTime="2026-09-10 23:41:21.216113888 +0000 UTC m=+296.683588225" Sep 10 23:41:24.950370 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:24.950338 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zkr6f_aacd921b-5c15-4f7f-8685-cfceebfe5407/console-operator/2.log" Sep 10 23:41:24.953032 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:24.953011 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zkr6f_aacd921b-5c15-4f7f-8685-cfceebfe5407/console-operator/2.log" Sep 10 23:41:24.960132 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:24.960113 2578 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 23:41:27.204752 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:27.204722 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-sxtjp" Sep 10 23:41:46.551251 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:46.551212 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f676b6c5d-z9t4c"] Sep 10 23:41:46.556590 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:46.556566 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f676b6c5d-z9t4c" Sep 10 23:41:46.561606 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:46.561583 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 23:41:46.561722 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:46.561594 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 23:41:46.561961 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:46.561923 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-x5ggk\"" Sep 10 23:41:46.562133 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:46.562117 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 23:41:46.562191 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:46.562157 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 23:41:46.575492 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:46.575471 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f676b6c5d-z9t4c"] Sep 10 23:41:46.672008 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:46.671972 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/846a18d9-8a3f-4e26-abbc-6d39a681cd3d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f676b6c5d-z9t4c\" (UID: \"846a18d9-8a3f-4e26-abbc-6d39a681cd3d\") " pod="opendatahub/opendatahub-operator-controller-manager-f676b6c5d-z9t4c" Sep 10 23:41:46.672008 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:46.672019 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/846a18d9-8a3f-4e26-abbc-6d39a681cd3d-webhook-cert\") pod \"opendatahub-operator-controller-manager-f676b6c5d-z9t4c\" (UID: \"846a18d9-8a3f-4e26-abbc-6d39a681cd3d\") " pod="opendatahub/opendatahub-operator-controller-manager-f676b6c5d-z9t4c" Sep 10 23:41:46.672231 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:46.672042 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/846a18d9-8a3f-4e26-abbc-6d39a681cd3d-tmp\") pod \"opendatahub-operator-controller-manager-f676b6c5d-z9t4c\" (UID: \"846a18d9-8a3f-4e26-abbc-6d39a681cd3d\") " pod="opendatahub/opendatahub-operator-controller-manager-f676b6c5d-z9t4c" Sep 10 23:41:46.672231 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:46.672120 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7czj\" (UniqueName: \"kubernetes.io/projected/846a18d9-8a3f-4e26-abbc-6d39a681cd3d-kube-api-access-p7czj\") pod \"opendatahub-operator-controller-manager-f676b6c5d-z9t4c\" (UID: \"846a18d9-8a3f-4e26-abbc-6d39a681cd3d\") " pod="opendatahub/opendatahub-operator-controller-manager-f676b6c5d-z9t4c" Sep 10 23:41:46.672231 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:46.672154 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/846a18d9-8a3f-4e26-abbc-6d39a681cd3d-manifests\") pod \"opendatahub-operator-controller-manager-f676b6c5d-z9t4c\" (UID: \"846a18d9-8a3f-4e26-abbc-6d39a681cd3d\") " pod="opendatahub/opendatahub-operator-controller-manager-f676b6c5d-z9t4c" Sep 10 23:41:46.773100 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:46.773049 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7czj\" (UniqueName: \"kubernetes.io/projected/846a18d9-8a3f-4e26-abbc-6d39a681cd3d-kube-api-access-p7czj\") pod \"opendatahub-operator-controller-manager-f676b6c5d-z9t4c\" (UID: \"846a18d9-8a3f-4e26-abbc-6d39a681cd3d\") " pod="opendatahub/opendatahub-operator-controller-manager-f676b6c5d-z9t4c" Sep 10 23:41:46.773298 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:46.773108 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/846a18d9-8a3f-4e26-abbc-6d39a681cd3d-manifests\") pod \"opendatahub-operator-controller-manager-f676b6c5d-z9t4c\" (UID: \"846a18d9-8a3f-4e26-abbc-6d39a681cd3d\") " pod="opendatahub/opendatahub-operator-controller-manager-f676b6c5d-z9t4c" Sep 10 23:41:46.773298 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:46.773162 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/846a18d9-8a3f-4e26-abbc-6d39a681cd3d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f676b6c5d-z9t4c\" (UID: \"846a18d9-8a3f-4e26-abbc-6d39a681cd3d\") " pod="opendatahub/opendatahub-operator-controller-manager-f676b6c5d-z9t4c" Sep 10 23:41:46.773298 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:46.773195 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/846a18d9-8a3f-4e26-abbc-6d39a681cd3d-webhook-cert\") pod \"opendatahub-operator-controller-manager-f676b6c5d-z9t4c\" (UID: \"846a18d9-8a3f-4e26-abbc-6d39a681cd3d\") " pod="opendatahub/opendatahub-operator-controller-manager-f676b6c5d-z9t4c" Sep 10 23:41:46.773298 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:46.773230 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/846a18d9-8a3f-4e26-abbc-6d39a681cd3d-tmp\") pod \"opendatahub-operator-controller-manager-f676b6c5d-z9t4c\" (UID: \"846a18d9-8a3f-4e26-abbc-6d39a681cd3d\") " pod="opendatahub/opendatahub-operator-controller-manager-f676b6c5d-z9t4c" Sep 10 23:41:46.773579 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:46.773541 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/846a18d9-8a3f-4e26-abbc-6d39a681cd3d-manifests\") pod \"opendatahub-operator-controller-manager-f676b6c5d-z9t4c\" (UID: \"846a18d9-8a3f-4e26-abbc-6d39a681cd3d\") " pod="opendatahub/opendatahub-operator-controller-manager-f676b6c5d-z9t4c" Sep 10 23:41:46.775516 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:46.775495 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/846a18d9-8a3f-4e26-abbc-6d39a681cd3d-tmp\") pod \"opendatahub-operator-controller-manager-f676b6c5d-z9t4c\" (UID: \"846a18d9-8a3f-4e26-abbc-6d39a681cd3d\") " pod="opendatahub/opendatahub-operator-controller-manager-f676b6c5d-z9t4c" Sep 10 23:41:46.775664 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:46.775646 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/846a18d9-8a3f-4e26-abbc-6d39a681cd3d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f676b6c5d-z9t4c\" (UID: \"846a18d9-8a3f-4e26-abbc-6d39a681cd3d\") " pod="opendatahub/opendatahub-operator-controller-manager-f676b6c5d-z9t4c" Sep 10 23:41:46.775822 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:46.775802 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/846a18d9-8a3f-4e26-abbc-6d39a681cd3d-webhook-cert\") pod \"opendatahub-operator-controller-manager-f676b6c5d-z9t4c\" (UID: \"846a18d9-8a3f-4e26-abbc-6d39a681cd3d\") " pod="opendatahub/opendatahub-operator-controller-manager-f676b6c5d-z9t4c" Sep 10 23:41:46.784859 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:46.784833 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7czj\" (UniqueName: \"kubernetes.io/projected/846a18d9-8a3f-4e26-abbc-6d39a681cd3d-kube-api-access-p7czj\") pod \"opendatahub-operator-controller-manager-f676b6c5d-z9t4c\" (UID: \"846a18d9-8a3f-4e26-abbc-6d39a681cd3d\") " pod="opendatahub/opendatahub-operator-controller-manager-f676b6c5d-z9t4c" Sep 10 23:41:46.866637 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:46.866551 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f676b6c5d-z9t4c" Sep 10 23:41:47.162128 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:47.162097 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f676b6c5d-z9t4c"] Sep 10 23:41:47.166500 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:41:47.166477 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846a18d9_8a3f_4e26_abbc_6d39a681cd3d.slice/crio-c03cee97424dfee56f655061ba0e536975e2aac7b51f7cc1883de2d081dd36f8 WatchSource:0}: Error finding container c03cee97424dfee56f655061ba0e536975e2aac7b51f7cc1883de2d081dd36f8: Status 404 returned error can't find the container with id c03cee97424dfee56f655061ba0e536975e2aac7b51f7cc1883de2d081dd36f8 Sep 10 23:41:47.168314 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:47.168292 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 23:41:47.289014 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:47.288976 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f676b6c5d-z9t4c" event={"ID":"846a18d9-8a3f-4e26-abbc-6d39a681cd3d","Type":"ContainerStarted","Data":"c03cee97424dfee56f655061ba0e536975e2aac7b51f7cc1883de2d081dd36f8"} Sep 10 23:41:50.303129 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:50.303047 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f676b6c5d-z9t4c" event={"ID":"846a18d9-8a3f-4e26-abbc-6d39a681cd3d","Type":"ContainerStarted","Data":"7c9f6705d37050aae364416a16815d1507f39b26018dd8ea944eefe032ac1f72"} Sep 10 23:41:51.308464 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:51.308369 2578 generic.go:356] "Generic (PLEG): container finished" podID="846a18d9-8a3f-4e26-abbc-6d39a681cd3d" containerID="7c9f6705d37050aae364416a16815d1507f39b26018dd8ea944eefe032ac1f72" exitCode=0 Sep 10 23:41:51.308826 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:51.308456 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f676b6c5d-z9t4c" event={"ID":"846a18d9-8a3f-4e26-abbc-6d39a681cd3d","Type":"ContainerDied","Data":"7c9f6705d37050aae364416a16815d1507f39b26018dd8ea944eefe032ac1f72"} Sep 10 23:41:52.314496 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:52.314461 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f676b6c5d-z9t4c" event={"ID":"846a18d9-8a3f-4e26-abbc-6d39a681cd3d","Type":"ContainerStarted","Data":"a745b87109025a3eea225413ac47c18bb709ae12819472dfc759f7ba5d177026"} Sep 10 23:41:52.314887 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:52.314574 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-f676b6c5d-z9t4c" Sep 10 23:41:52.377328 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:41:52.377280 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-f676b6c5d-z9t4c" podStartSLOduration=2.234437069 podStartE2EDuration="6.377261313s" totalImagesPullingTime="3.083770523s" totalInitContainerRuntime="1.059053721s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:41:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:41:52.374713579 +0000 UTC m=+327.842187914" watchObservedRunningTime="2026-09-10 23:41:52.377261313 +0000 UTC m=+327.844735643" Sep 10 23:42:03.324612 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:03.324165 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-f676b6c5d-z9t4c" Sep 10 23:42:04.333027 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:04.332993 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-f59994447-lbf89"] Sep 10 23:42:04.341037 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:04.341014 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-f59994447-lbf89" Sep 10 23:42:04.344214 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:04.344193 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 10 23:42:04.344332 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:04.344221 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 10 23:42:04.344332 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:04.344249 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 10 23:42:04.344332 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:04.344215 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 10 23:42:04.344332 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:04.344282 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-bllhr\"" Sep 10 23:42:04.344332 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:04.344198 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 10 23:42:04.349031 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:04.349013 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-f59994447-lbf89"] Sep 10 23:42:04.400401 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:04.400380 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b281584-acde-4df9-8810-37383aaf00d7-metrics-cert\") pod \"lws-controller-manager-f59994447-lbf89\" (UID: \"9b281584-acde-4df9-8810-37383aaf00d7\") " pod="openshift-lws-operator/lws-controller-manager-f59994447-lbf89" Sep 10 23:42:04.400529 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:04.400406 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtsl4\" (UniqueName: \"kubernetes.io/projected/9b281584-acde-4df9-8810-37383aaf00d7-kube-api-access-vtsl4\") pod \"lws-controller-manager-f59994447-lbf89\" (UID: \"9b281584-acde-4df9-8810-37383aaf00d7\") " pod="openshift-lws-operator/lws-controller-manager-f59994447-lbf89" Sep 10 23:42:04.400529 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:04.400426 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9b281584-acde-4df9-8810-37383aaf00d7-manager-config\") pod \"lws-controller-manager-f59994447-lbf89\" (UID: \"9b281584-acde-4df9-8810-37383aaf00d7\") " pod="openshift-lws-operator/lws-controller-manager-f59994447-lbf89" Sep 10 23:42:04.400529 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:04.400444 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b281584-acde-4df9-8810-37383aaf00d7-cert\") pod \"lws-controller-manager-f59994447-lbf89\" (UID: \"9b281584-acde-4df9-8810-37383aaf00d7\") " pod="openshift-lws-operator/lws-controller-manager-f59994447-lbf89" Sep 10 23:42:04.501264 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:04.501236 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b281584-acde-4df9-8810-37383aaf00d7-metrics-cert\") pod \"lws-controller-manager-f59994447-lbf89\" (UID: \"9b281584-acde-4df9-8810-37383aaf00d7\") " pod="openshift-lws-operator/lws-controller-manager-f59994447-lbf89" Sep 10 23:42:04.501360 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:04.501268 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtsl4\" (UniqueName: \"kubernetes.io/projected/9b281584-acde-4df9-8810-37383aaf00d7-kube-api-access-vtsl4\") pod \"lws-controller-manager-f59994447-lbf89\" (UID: \"9b281584-acde-4df9-8810-37383aaf00d7\") " pod="openshift-lws-operator/lws-controller-manager-f59994447-lbf89" Sep 10 23:42:04.501360 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:04.501293 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9b281584-acde-4df9-8810-37383aaf00d7-manager-config\") pod \"lws-controller-manager-f59994447-lbf89\" (UID: \"9b281584-acde-4df9-8810-37383aaf00d7\") " pod="openshift-lws-operator/lws-controller-manager-f59994447-lbf89" Sep 10 23:42:04.501360 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:04.501308 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b281584-acde-4df9-8810-37383aaf00d7-cert\") pod \"lws-controller-manager-f59994447-lbf89\" (UID: \"9b281584-acde-4df9-8810-37383aaf00d7\") " pod="openshift-lws-operator/lws-controller-manager-f59994447-lbf89" Sep 10 23:42:04.501866 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:04.501844 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9b281584-acde-4df9-8810-37383aaf00d7-manager-config\") pod \"lws-controller-manager-f59994447-lbf89\" (UID: \"9b281584-acde-4df9-8810-37383aaf00d7\") " pod="openshift-lws-operator/lws-controller-manager-f59994447-lbf89" Sep 10 23:42:04.503693 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:04.503671 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b281584-acde-4df9-8810-37383aaf00d7-metrics-cert\") pod \"lws-controller-manager-f59994447-lbf89\" (UID: \"9b281584-acde-4df9-8810-37383aaf00d7\") " pod="openshift-lws-operator/lws-controller-manager-f59994447-lbf89" Sep 10 23:42:04.503766 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:04.503719 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b281584-acde-4df9-8810-37383aaf00d7-cert\") pod \"lws-controller-manager-f59994447-lbf89\" (UID: \"9b281584-acde-4df9-8810-37383aaf00d7\") " pod="openshift-lws-operator/lws-controller-manager-f59994447-lbf89" Sep 10 23:42:04.509656 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:04.509628 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtsl4\" (UniqueName: \"kubernetes.io/projected/9b281584-acde-4df9-8810-37383aaf00d7-kube-api-access-vtsl4\") pod \"lws-controller-manager-f59994447-lbf89\" (UID: \"9b281584-acde-4df9-8810-37383aaf00d7\") " pod="openshift-lws-operator/lws-controller-manager-f59994447-lbf89" Sep 10 23:42:04.649945 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:04.649904 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-f59994447-lbf89" Sep 10 23:42:04.933105 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:04.933016 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-f59994447-lbf89"] Sep 10 23:42:04.936021 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:42:04.935993 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b281584_acde_4df9_8810_37383aaf00d7.slice/crio-0b05b145b87e1b9f49b3835ca3c245a99f53fc658a68c90172d7fed3e4b6c667 WatchSource:0}: Error finding container 0b05b145b87e1b9f49b3835ca3c245a99f53fc658a68c90172d7fed3e4b6c667: Status 404 returned error can't find the container with id 0b05b145b87e1b9f49b3835ca3c245a99f53fc658a68c90172d7fed3e4b6c667 Sep 10 23:42:05.365329 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:05.365255 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-f59994447-lbf89" event={"ID":"9b281584-acde-4df9-8810-37383aaf00d7","Type":"ContainerStarted","Data":"0b05b145b87e1b9f49b3835ca3c245a99f53fc658a68c90172d7fed3e4b6c667"} Sep 10 23:42:08.378067 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:08.378025 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-f59994447-lbf89" event={"ID":"9b281584-acde-4df9-8810-37383aaf00d7","Type":"ContainerStarted","Data":"78b371114ec49a2ada30923f610ee2d42103ae51544d4548c22876914955bfcd"} Sep 10 23:42:08.378477 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:08.378156 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-f59994447-lbf89" Sep 10 23:42:08.400700 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:08.400648 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-f59994447-lbf89" podStartSLOduration=1.9945609439999998 podStartE2EDuration="4.400628382s" totalImagesPullingTime="2.406067438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:42:08.400578977 +0000 UTC m=+343.868053306" watchObservedRunningTime="2026-09-10 23:42:08.400628382 +0000 UTC m=+343.868102708" Sep 10 23:42:15.278447 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.278414 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m"] Sep 10 23:42:15.286023 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.285011 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.288078 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.288055 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 23:42:15.288265 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.288114 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-t9jd9\"" Sep 10 23:42:15.288389 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.288121 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 23:42:15.288389 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.288131 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 23:42:15.294089 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.294068 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m"] Sep 10 23:42:15.387479 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.387443 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a705061-7770-4fae-a453-dd7491b000e6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m\" (UID: \"9a705061-7770-4fae-a453-dd7491b000e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.387618 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.387486 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a705061-7770-4fae-a453-dd7491b000e6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m\" (UID: \"9a705061-7770-4fae-a453-dd7491b000e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.387618 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.387533 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a705061-7770-4fae-a453-dd7491b000e6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m\" (UID: \"9a705061-7770-4fae-a453-dd7491b000e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.387618 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.387568 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a705061-7770-4fae-a453-dd7491b000e6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m\" (UID: \"9a705061-7770-4fae-a453-dd7491b000e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.387618 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.387590 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a705061-7770-4fae-a453-dd7491b000e6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m\" (UID: \"9a705061-7770-4fae-a453-dd7491b000e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.387773 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.387628 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a705061-7770-4fae-a453-dd7491b000e6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m\" (UID: \"9a705061-7770-4fae-a453-dd7491b000e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.387773 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.387652 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a705061-7770-4fae-a453-dd7491b000e6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m\" (UID: \"9a705061-7770-4fae-a453-dd7491b000e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.387773 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.387674 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a705061-7770-4fae-a453-dd7491b000e6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m\" (UID: \"9a705061-7770-4fae-a453-dd7491b000e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.387773 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.387701 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr77n\" (UniqueName: \"kubernetes.io/projected/9a705061-7770-4fae-a453-dd7491b000e6-kube-api-access-zr77n\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m\" (UID: \"9a705061-7770-4fae-a453-dd7491b000e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.489048 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.489016 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a705061-7770-4fae-a453-dd7491b000e6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m\" (UID: \"9a705061-7770-4fae-a453-dd7491b000e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.489048 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.489050 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a705061-7770-4fae-a453-dd7491b000e6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m\" (UID: \"9a705061-7770-4fae-a453-dd7491b000e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.489256 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.489077 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr77n\" (UniqueName: \"kubernetes.io/projected/9a705061-7770-4fae-a453-dd7491b000e6-kube-api-access-zr77n\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m\" (UID: \"9a705061-7770-4fae-a453-dd7491b000e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.489256 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.489111 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a705061-7770-4fae-a453-dd7491b000e6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m\" (UID: \"9a705061-7770-4fae-a453-dd7491b000e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.489256 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.489136 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a705061-7770-4fae-a453-dd7491b000e6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m\" (UID: \"9a705061-7770-4fae-a453-dd7491b000e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.489256 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.489162 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a705061-7770-4fae-a453-dd7491b000e6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m\" (UID: \"9a705061-7770-4fae-a453-dd7491b000e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.489464 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.489271 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a705061-7770-4fae-a453-dd7491b000e6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m\" (UID: \"9a705061-7770-4fae-a453-dd7491b000e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.489464 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.489320 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a705061-7770-4fae-a453-dd7491b000e6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m\" (UID: \"9a705061-7770-4fae-a453-dd7491b000e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.489464 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.489381 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a705061-7770-4fae-a453-dd7491b000e6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m\" (UID: \"9a705061-7770-4fae-a453-dd7491b000e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.489464 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.489445 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a705061-7770-4fae-a453-dd7491b000e6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m\" (UID: \"9a705061-7770-4fae-a453-dd7491b000e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.489663 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.489636 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a705061-7770-4fae-a453-dd7491b000e6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m\" (UID: \"9a705061-7770-4fae-a453-dd7491b000e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.489663 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.489654 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a705061-7770-4fae-a453-dd7491b000e6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m\" (UID: \"9a705061-7770-4fae-a453-dd7491b000e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.489768 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.489723 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a705061-7770-4fae-a453-dd7491b000e6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m\" (UID: \"9a705061-7770-4fae-a453-dd7491b000e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.489854 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.489833 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a705061-7770-4fae-a453-dd7491b000e6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m\" (UID: \"9a705061-7770-4fae-a453-dd7491b000e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.491417 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.491389 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a705061-7770-4fae-a453-dd7491b000e6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m\" (UID: \"9a705061-7770-4fae-a453-dd7491b000e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.491759 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.491741 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a705061-7770-4fae-a453-dd7491b000e6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m\" (UID: \"9a705061-7770-4fae-a453-dd7491b000e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.503724 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.503694 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a705061-7770-4fae-a453-dd7491b000e6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m\" (UID: \"9a705061-7770-4fae-a453-dd7491b000e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.504343 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.504325 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr77n\" (UniqueName: \"kubernetes.io/projected/9a705061-7770-4fae-a453-dd7491b000e6-kube-api-access-zr77n\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m\" (UID: \"9a705061-7770-4fae-a453-dd7491b000e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.597519 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.597442 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:15.875289 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:15.875266 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m"] Sep 10 23:42:15.877774 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:42:15.877746 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a705061_7770_4fae_a453_dd7491b000e6.slice/crio-58a90c7213718e8ad86443ceef3c8c820e35d5c989eceb83768627c732a8e9b5 WatchSource:0}: Error finding container 58a90c7213718e8ad86443ceef3c8c820e35d5c989eceb83768627c732a8e9b5: Status 404 returned error can't find the container with id 58a90c7213718e8ad86443ceef3c8c820e35d5c989eceb83768627c732a8e9b5 Sep 10 23:42:16.412304 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:16.412232 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" event={"ID":"9a705061-7770-4fae-a453-dd7491b000e6","Type":"ContainerStarted","Data":"58a90c7213718e8ad86443ceef3c8c820e35d5c989eceb83768627c732a8e9b5"} Sep 10 23:42:16.488369 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:16.488329 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-94j5h"] Sep 10 23:42:16.493808 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:16.493694 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-94j5h" Sep 10 23:42:16.497614 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:16.497361 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 10 23:42:16.497614 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:16.497374 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-q27fj\"" Sep 10 23:42:16.501320 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:16.501262 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-94j5h"] Sep 10 23:42:16.597725 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:16.597688 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8860c823-1800-441d-8c50-4c10e31e99aa-cert\") pod \"odh-model-controller-7998c67dc4-94j5h\" (UID: \"8860c823-1800-441d-8c50-4c10e31e99aa\") " pod="opendatahub/odh-model-controller-7998c67dc4-94j5h" Sep 10 23:42:16.597957 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:16.597784 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdj62\" (UniqueName: \"kubernetes.io/projected/8860c823-1800-441d-8c50-4c10e31e99aa-kube-api-access-wdj62\") pod \"odh-model-controller-7998c67dc4-94j5h\" (UID: \"8860c823-1800-441d-8c50-4c10e31e99aa\") " pod="opendatahub/odh-model-controller-7998c67dc4-94j5h" Sep 10 23:42:16.698603 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:16.698495 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdj62\" (UniqueName: \"kubernetes.io/projected/8860c823-1800-441d-8c50-4c10e31e99aa-kube-api-access-wdj62\") pod \"odh-model-controller-7998c67dc4-94j5h\" (UID: \"8860c823-1800-441d-8c50-4c10e31e99aa\") " pod="opendatahub/odh-model-controller-7998c67dc4-94j5h" Sep 10 23:42:16.698603 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:16.698549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8860c823-1800-441d-8c50-4c10e31e99aa-cert\") pod \"odh-model-controller-7998c67dc4-94j5h\" (UID: \"8860c823-1800-441d-8c50-4c10e31e99aa\") " pod="opendatahub/odh-model-controller-7998c67dc4-94j5h" Sep 10 23:42:16.698859 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:42:16.698676 2578 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 10 23:42:16.698859 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:42:16.698732 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8860c823-1800-441d-8c50-4c10e31e99aa-cert podName:8860c823-1800-441d-8c50-4c10e31e99aa nodeName:}" failed. No retries permitted until 2026-09-10 23:42:17.198713927 +0000 UTC m=+352.666188246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8860c823-1800-441d-8c50-4c10e31e99aa-cert") pod "odh-model-controller-7998c67dc4-94j5h" (UID: "8860c823-1800-441d-8c50-4c10e31e99aa") : secret "odh-model-controller-webhook-cert" not found Sep 10 23:42:16.709731 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:16.709700 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdj62\" (UniqueName: \"kubernetes.io/projected/8860c823-1800-441d-8c50-4c10e31e99aa-kube-api-access-wdj62\") pod \"odh-model-controller-7998c67dc4-94j5h\" (UID: \"8860c823-1800-441d-8c50-4c10e31e99aa\") " pod="opendatahub/odh-model-controller-7998c67dc4-94j5h" Sep 10 23:42:17.202878 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:17.202786 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8860c823-1800-441d-8c50-4c10e31e99aa-cert\") pod \"odh-model-controller-7998c67dc4-94j5h\" (UID: \"8860c823-1800-441d-8c50-4c10e31e99aa\") " pod="opendatahub/odh-model-controller-7998c67dc4-94j5h" Sep 10 23:42:17.205730 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:17.205694 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8860c823-1800-441d-8c50-4c10e31e99aa-cert\") pod \"odh-model-controller-7998c67dc4-94j5h\" (UID: \"8860c823-1800-441d-8c50-4c10e31e99aa\") " pod="opendatahub/odh-model-controller-7998c67dc4-94j5h" Sep 10 23:42:17.420347 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:17.420313 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-94j5h" Sep 10 23:42:18.409082 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:18.409039 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:42:18.409188 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:18.409139 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:42:18.409226 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:18.409185 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:42:18.609001 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:18.608899 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-94j5h"] Sep 10 23:42:18.612834 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:42:18.612802 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8860c823_1800_441d_8c50_4c10e31e99aa.slice/crio-35aeb34b3e7d5cfc163a815b11749643bd11230a5e1ac5b3c9c32a3d8c671695 WatchSource:0}: Error finding container 35aeb34b3e7d5cfc163a815b11749643bd11230a5e1ac5b3c9c32a3d8c671695: Status 404 returned error can't find the container with id 35aeb34b3e7d5cfc163a815b11749643bd11230a5e1ac5b3c9c32a3d8c671695 Sep 10 23:42:19.384701 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:19.384023 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-f59994447-lbf89" Sep 10 23:42:19.427034 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:19.426982 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-94j5h" event={"ID":"8860c823-1800-441d-8c50-4c10e31e99aa","Type":"ContainerStarted","Data":"35aeb34b3e7d5cfc163a815b11749643bd11230a5e1ac5b3c9c32a3d8c671695"} Sep 10 23:42:19.430272 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:19.430231 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" event={"ID":"9a705061-7770-4fae-a453-dd7491b000e6","Type":"ContainerStarted","Data":"e6a3b51b56432a49c5a3e7cfd20e806fb861fc51cf5491692ead94fa21282b08"} Sep 10 23:42:19.451631 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:19.451374 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" podStartSLOduration=1.922457511 podStartE2EDuration="4.4513539s" totalImagesPullingTime="2.528896389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:42:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:42:19.451246176 +0000 UTC m=+354.918720528" watchObservedRunningTime="2026-09-10 23:42:19.4513539 +0000 UTC m=+354.918828242" Sep 10 23:42:19.597957 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:19.597899 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:19.603708 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:19.603679 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:20.435743 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:20.435711 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:20.436956 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:20.436912 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m" Sep 10 23:42:21.442977 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:21.442923 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-94j5h" event={"ID":"8860c823-1800-441d-8c50-4c10e31e99aa","Type":"ContainerStarted","Data":"4412ba8cdc893b1522cbd70ed7c246ba357d2a10510b1afa09379c665b85bff0"} Sep 10 23:42:21.443390 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:21.443354 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-94j5h" Sep 10 23:42:21.460746 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:21.460690 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-94j5h" podStartSLOduration=2.720751101 podStartE2EDuration="5.460672269s" totalImagesPullingTime="2.739921168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:42:21.458439268 +0000 UTC m=+356.925913594" watchObservedRunningTime="2026-09-10 23:42:21.460672269 +0000 UTC m=+356.928146595" Sep 10 23:42:24.286502 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.286466 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444"] Sep 10 23:42:24.289697 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.289680 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" Sep 10 23:42:24.292594 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.292569 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 10 23:42:24.292718 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.292622 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-psxj5\"" Sep 10 23:42:24.292831 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.292814 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 10 23:42:24.293074 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.293052 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 10 23:42:24.303684 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.303661 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444"] Sep 10 23:42:24.370136 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.370087 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4acfeb77-59ca-4083-8f4b-062f47030c56-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-hh444\" (UID: \"4acfeb77-59ca-4083-8f4b-062f47030c56\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" Sep 10 23:42:24.370302 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.370152 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4acfeb77-59ca-4083-8f4b-062f47030c56-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-hh444\" (UID: \"4acfeb77-59ca-4083-8f4b-062f47030c56\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" Sep 10 23:42:24.370302 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.370199 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4acfeb77-59ca-4083-8f4b-062f47030c56-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-hh444\" (UID: \"4acfeb77-59ca-4083-8f4b-062f47030c56\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" Sep 10 23:42:24.370302 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.370218 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4acfeb77-59ca-4083-8f4b-062f47030c56-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-hh444\" (UID: \"4acfeb77-59ca-4083-8f4b-062f47030c56\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" Sep 10 23:42:24.370489 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.370314 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4acfeb77-59ca-4083-8f4b-062f47030c56-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-hh444\" (UID: \"4acfeb77-59ca-4083-8f4b-062f47030c56\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" Sep 10 23:42:24.370489 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.370369 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4acfeb77-59ca-4083-8f4b-062f47030c56-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-hh444\" (UID: \"4acfeb77-59ca-4083-8f4b-062f47030c56\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" Sep 10 23:42:24.370489 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.370410 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fztjz\" (UniqueName: \"kubernetes.io/projected/4acfeb77-59ca-4083-8f4b-062f47030c56-kube-api-access-fztjz\") pod \"istiod-openshift-gateway-5677ccb49b-hh444\" (UID: \"4acfeb77-59ca-4083-8f4b-062f47030c56\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" Sep 10 23:42:24.471713 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.471680 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4acfeb77-59ca-4083-8f4b-062f47030c56-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-hh444\" (UID: \"4acfeb77-59ca-4083-8f4b-062f47030c56\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" Sep 10 23:42:24.471713 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.471717 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4acfeb77-59ca-4083-8f4b-062f47030c56-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-hh444\" (UID: \"4acfeb77-59ca-4083-8f4b-062f47030c56\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" Sep 10 23:42:24.472005 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.471740 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4acfeb77-59ca-4083-8f4b-062f47030c56-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-hh444\" (UID: \"4acfeb77-59ca-4083-8f4b-062f47030c56\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" Sep 10 23:42:24.472005 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.471767 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4acfeb77-59ca-4083-8f4b-062f47030c56-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-hh444\" (UID: \"4acfeb77-59ca-4083-8f4b-062f47030c56\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" Sep 10 23:42:24.472005 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.471790 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fztjz\" (UniqueName: \"kubernetes.io/projected/4acfeb77-59ca-4083-8f4b-062f47030c56-kube-api-access-fztjz\") pod \"istiod-openshift-gateway-5677ccb49b-hh444\" (UID: \"4acfeb77-59ca-4083-8f4b-062f47030c56\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" Sep 10 23:42:24.472005 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.471813 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4acfeb77-59ca-4083-8f4b-062f47030c56-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-hh444\" (UID: \"4acfeb77-59ca-4083-8f4b-062f47030c56\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" Sep 10 23:42:24.472005 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.471834 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4acfeb77-59ca-4083-8f4b-062f47030c56-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-hh444\" (UID: \"4acfeb77-59ca-4083-8f4b-062f47030c56\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" Sep 10 23:42:24.472449 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.472427 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4acfeb77-59ca-4083-8f4b-062f47030c56-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-hh444\" (UID: \"4acfeb77-59ca-4083-8f4b-062f47030c56\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" Sep 10 23:42:24.474284 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.474262 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4acfeb77-59ca-4083-8f4b-062f47030c56-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-hh444\" (UID: \"4acfeb77-59ca-4083-8f4b-062f47030c56\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" Sep 10 23:42:24.474413 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.474391 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4acfeb77-59ca-4083-8f4b-062f47030c56-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-hh444\" (UID: \"4acfeb77-59ca-4083-8f4b-062f47030c56\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" Sep 10 23:42:24.474712 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.474690 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4acfeb77-59ca-4083-8f4b-062f47030c56-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-hh444\" (UID: \"4acfeb77-59ca-4083-8f4b-062f47030c56\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" Sep 10 23:42:24.474810 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.474744 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4acfeb77-59ca-4083-8f4b-062f47030c56-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-hh444\" (UID: \"4acfeb77-59ca-4083-8f4b-062f47030c56\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" Sep 10 23:42:24.479818 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.479795 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4acfeb77-59ca-4083-8f4b-062f47030c56-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-hh444\" (UID: \"4acfeb77-59ca-4083-8f4b-062f47030c56\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" Sep 10 23:42:24.480074 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.480055 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fztjz\" (UniqueName: \"kubernetes.io/projected/4acfeb77-59ca-4083-8f4b-062f47030c56-kube-api-access-fztjz\") pod \"istiod-openshift-gateway-5677ccb49b-hh444\" (UID: \"4acfeb77-59ca-4083-8f4b-062f47030c56\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" Sep 10 23:42:24.605705 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.605628 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" Sep 10 23:42:24.895833 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:24.895806 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444"] Sep 10 23:42:24.898241 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:42:24.898213 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4acfeb77_59ca_4083_8f4b_062f47030c56.slice/crio-84aec6eac42d28d13f7bd478436d10ab17367717cfd44d8117c78f1f131d0876 WatchSource:0}: Error finding container 84aec6eac42d28d13f7bd478436d10ab17367717cfd44d8117c78f1f131d0876: Status 404 returned error can't find the container with id 84aec6eac42d28d13f7bd478436d10ab17367717cfd44d8117c78f1f131d0876 Sep 10 23:42:25.461108 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:25.461075 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" event={"ID":"4acfeb77-59ca-4083-8f4b-062f47030c56","Type":"ContainerStarted","Data":"84aec6eac42d28d13f7bd478436d10ab17367717cfd44d8117c78f1f131d0876"} Sep 10 23:42:27.851552 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:27.851521 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:42:27.851795 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:27.851582 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:42:28.475802 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:28.475762 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" event={"ID":"4acfeb77-59ca-4083-8f4b-062f47030c56","Type":"ContainerStarted","Data":"cdd3e97ed0c22d92f31f29b0e102489b2260114489314489a54b935fa55ff7d5"} Sep 10 23:42:28.476009 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:28.475958 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" Sep 10 23:42:28.477764 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:28.477737 2578 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-hh444 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 23:42:28.477858 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:28.477789 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" podUID="4acfeb77-59ca-4083-8f4b-062f47030c56" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 23:42:28.518144 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:28.518095 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" podStartSLOduration=1.567043272 podStartE2EDuration="4.51807854s" totalImagesPullingTime="2.951035268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:42:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:42:28.515335112 +0000 UTC m=+363.982809470" watchObservedRunningTime="2026-09-10 23:42:28.51807854 +0000 UTC m=+363.985552865" Sep 10 23:42:29.482028 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:29.481996 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hh444" Sep 10 23:42:32.450162 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:42:32.450131 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-94j5h" Sep 10 23:46:00.619300 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:00.619261 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-fkscd"] Sep 10 23:46:00.622451 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:00.622435 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fkscd" Sep 10 23:46:00.624623 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:00.624601 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 10 23:46:00.624739 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:00.624682 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-qq8gw\"" Sep 10 23:46:00.632811 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:00.632784 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-fkscd"] Sep 10 23:46:00.723620 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:00.723580 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b88901f-6268-423c-b4d2-8d5a44cdb607-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-fkscd\" (UID: \"2b88901f-6268-423c-b4d2-8d5a44cdb607\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fkscd" Sep 10 23:46:00.723620 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:00.723621 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s969\" (UniqueName: \"kubernetes.io/projected/2b88901f-6268-423c-b4d2-8d5a44cdb607-kube-api-access-4s969\") pod \"payload-pre-processing-59fbc47bf8-fkscd\" (UID: \"2b88901f-6268-423c-b4d2-8d5a44cdb607\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fkscd" Sep 10 23:46:00.824802 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:00.824772 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b88901f-6268-423c-b4d2-8d5a44cdb607-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-fkscd\" (UID: \"2b88901f-6268-423c-b4d2-8d5a44cdb607\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fkscd" Sep 10 23:46:00.824802 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:00.824806 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s969\" (UniqueName: \"kubernetes.io/projected/2b88901f-6268-423c-b4d2-8d5a44cdb607-kube-api-access-4s969\") pod \"payload-pre-processing-59fbc47bf8-fkscd\" (UID: \"2b88901f-6268-423c-b4d2-8d5a44cdb607\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fkscd" Sep 10 23:46:00.825490 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:00.825471 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b88901f-6268-423c-b4d2-8d5a44cdb607-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-fkscd\" (UID: \"2b88901f-6268-423c-b4d2-8d5a44cdb607\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fkscd" Sep 10 23:46:00.834887 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:00.834866 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s969\" (UniqueName: \"kubernetes.io/projected/2b88901f-6268-423c-b4d2-8d5a44cdb607-kube-api-access-4s969\") pod \"payload-pre-processing-59fbc47bf8-fkscd\" (UID: \"2b88901f-6268-423c-b4d2-8d5a44cdb607\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fkscd" Sep 10 23:46:00.933442 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:00.933383 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fkscd" Sep 10 23:46:01.201148 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:01.201064 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-fkscd"] Sep 10 23:46:01.203823 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:46:01.203797 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b88901f_6268_423c_b4d2_8d5a44cdb607.slice/crio-40bc865ea2ec15f734e5266f02cc1a64f5a935220899d3316458eb6d12decae8 WatchSource:0}: Error finding container 40bc865ea2ec15f734e5266f02cc1a64f5a935220899d3316458eb6d12decae8: Status 404 returned error can't find the container with id 40bc865ea2ec15f734e5266f02cc1a64f5a935220899d3316458eb6d12decae8 Sep 10 23:46:01.325108 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:01.325074 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fkscd" event={"ID":"2b88901f-6268-423c-b4d2-8d5a44cdb607","Type":"ContainerStarted","Data":"40bc865ea2ec15f734e5266f02cc1a64f5a935220899d3316458eb6d12decae8"} Sep 10 23:46:03.334755 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:03.334709 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fkscd" event={"ID":"2b88901f-6268-423c-b4d2-8d5a44cdb607","Type":"ContainerStarted","Data":"b2f75aeee2bd2de6972ff69fdc43d508f51920f68337ce2e8ecd9dcc4794d889"} Sep 10 23:46:03.335185 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:03.334798 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fkscd" Sep 10 23:46:03.351988 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:03.351914 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fkscd" podStartSLOduration=1.896217944 podStartE2EDuration="3.351896513s" totalImagesPullingTime="1.455678569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:46:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:46:03.351495855 +0000 UTC m=+578.818970180" watchObservedRunningTime="2026-09-10 23:46:03.351896513 +0000 UTC m=+578.819370837" Sep 10 23:46:14.341083 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:14.341039 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fkscd" Sep 10 23:46:20.545107 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:20.545076 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6"] Sep 10 23:46:20.548148 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:20.548131 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6" Sep 10 23:46:20.555884 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:20.555838 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 10 23:46:20.556470 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:20.556452 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 23:46:20.556916 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:20.556893 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 23:46:20.559762 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:20.559744 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rg6w4\"" Sep 10 23:46:20.575755 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:20.575716 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6"] Sep 10 23:46:20.703306 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:20.703271 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2efb490-ac69-4c42-a41e-7c98483436f8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6\" (UID: \"d2efb490-ac69-4c42-a41e-7c98483436f8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6" Sep 10 23:46:20.703478 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:20.703324 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2efb490-ac69-4c42-a41e-7c98483436f8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6\" (UID: \"d2efb490-ac69-4c42-a41e-7c98483436f8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6" Sep 10 23:46:20.703478 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:20.703377 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvp2n\" (UniqueName: \"kubernetes.io/projected/d2efb490-ac69-4c42-a41e-7c98483436f8-kube-api-access-gvp2n\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6\" (UID: \"d2efb490-ac69-4c42-a41e-7c98483436f8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6" Sep 10 23:46:20.703478 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:20.703401 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2efb490-ac69-4c42-a41e-7c98483436f8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6\" (UID: \"d2efb490-ac69-4c42-a41e-7c98483436f8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6" Sep 10 23:46:20.703478 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:20.703455 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2efb490-ac69-4c42-a41e-7c98483436f8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6\" (UID: \"d2efb490-ac69-4c42-a41e-7c98483436f8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6" Sep 10 23:46:20.703636 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:20.703485 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2efb490-ac69-4c42-a41e-7c98483436f8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6\" (UID: \"d2efb490-ac69-4c42-a41e-7c98483436f8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6" Sep 10 23:46:20.804223 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:20.804120 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2efb490-ac69-4c42-a41e-7c98483436f8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6\" (UID: \"d2efb490-ac69-4c42-a41e-7c98483436f8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6" Sep 10 23:46:20.804223 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:20.804215 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2efb490-ac69-4c42-a41e-7c98483436f8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6\" (UID: \"d2efb490-ac69-4c42-a41e-7c98483436f8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6" Sep 10 23:46:20.804456 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:20.804248 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvp2n\" (UniqueName: \"kubernetes.io/projected/d2efb490-ac69-4c42-a41e-7c98483436f8-kube-api-access-gvp2n\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6\" (UID: \"d2efb490-ac69-4c42-a41e-7c98483436f8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6" Sep 10 23:46:20.804456 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:20.804287 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2efb490-ac69-4c42-a41e-7c98483436f8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6\" (UID: \"d2efb490-ac69-4c42-a41e-7c98483436f8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6" Sep 10 23:46:20.804456 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:20.804319 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2efb490-ac69-4c42-a41e-7c98483436f8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6\" (UID: \"d2efb490-ac69-4c42-a41e-7c98483436f8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6" Sep 10 23:46:20.804456 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:20.804341 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2efb490-ac69-4c42-a41e-7c98483436f8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6\" (UID: \"d2efb490-ac69-4c42-a41e-7c98483436f8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6" Sep 10 23:46:20.804770 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:20.804739 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2efb490-ac69-4c42-a41e-7c98483436f8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6\" (UID: \"d2efb490-ac69-4c42-a41e-7c98483436f8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6" Sep 10 23:46:20.804901 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:20.804739 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2efb490-ac69-4c42-a41e-7c98483436f8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6\" (UID: \"d2efb490-ac69-4c42-a41e-7c98483436f8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6" Sep 10 23:46:20.804901 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:20.804850 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2efb490-ac69-4c42-a41e-7c98483436f8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6\" (UID: \"d2efb490-ac69-4c42-a41e-7c98483436f8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6" Sep 10 23:46:20.806496 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:20.806471 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2efb490-ac69-4c42-a41e-7c98483436f8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6\" (UID: \"d2efb490-ac69-4c42-a41e-7c98483436f8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6" Sep 10 23:46:20.806792 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:20.806774 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2efb490-ac69-4c42-a41e-7c98483436f8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6\" (UID: \"d2efb490-ac69-4c42-a41e-7c98483436f8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6" Sep 10 23:46:20.811876 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:20.811854 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvp2n\" (UniqueName: \"kubernetes.io/projected/d2efb490-ac69-4c42-a41e-7c98483436f8-kube-api-access-gvp2n\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6\" (UID: \"d2efb490-ac69-4c42-a41e-7c98483436f8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6" Sep 10 23:46:20.857894 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:20.857851 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6" Sep 10 23:46:21.128612 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:21.128565 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6"] Sep 10 23:46:21.131346 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:46:21.131316 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2efb490_ac69_4c42_a41e_7c98483436f8.slice/crio-72d4545c28a2b6ba1cd44949a82e374746fb46983f5f8585898eedbfa38e5745 WatchSource:0}: Error finding container 72d4545c28a2b6ba1cd44949a82e374746fb46983f5f8585898eedbfa38e5745: Status 404 returned error can't find the container with id 72d4545c28a2b6ba1cd44949a82e374746fb46983f5f8585898eedbfa38e5745 Sep 10 23:46:21.410976 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:21.410865 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6" event={"ID":"d2efb490-ac69-4c42-a41e-7c98483436f8","Type":"ContainerStarted","Data":"72d4545c28a2b6ba1cd44949a82e374746fb46983f5f8585898eedbfa38e5745"} Sep 10 23:46:22.416113 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:22.416073 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6" event={"ID":"d2efb490-ac69-4c42-a41e-7c98483436f8","Type":"ContainerStarted","Data":"c75b50dde7149ee96d9258df0ae400d4903034aa5e8f51f2984a8c04d6ca0cf6"} Sep 10 23:46:22.431963 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:22.431902 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6" podStartSLOduration=1.478805051 podStartE2EDuration="2.431888357s" totalImagesPullingTime="953.083306ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:46:22.431124949 +0000 UTC m=+597.898599274" watchObservedRunningTime="2026-09-10 23:46:22.431888357 +0000 UTC m=+597.899362681" Sep 10 23:46:24.976354 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:24.976325 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zkr6f_aacd921b-5c15-4f7f-8685-cfceebfe5407/console-operator/2.log" Sep 10 23:46:24.976744 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:24.976335 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zkr6f_aacd921b-5c15-4f7f-8685-cfceebfe5407/console-operator/2.log" Sep 10 23:46:30.858489 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:30.858442 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6" Sep 10 23:46:30.860865 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:30.858732 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6" Sep 10 23:46:30.871404 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:30.871381 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6" Sep 10 23:46:31.464375 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:31.464345 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6" Sep 10 23:46:40.020247 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:40.020211 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb"] Sep 10 23:46:40.023360 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:40.023342 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb" Sep 10 23:46:40.025595 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:40.025574 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 10 23:46:40.033722 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:40.033697 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb"] Sep 10 23:46:40.070318 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:40.070291 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42570555-7956-4ee2-9c38-0c86484711ef-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb\" (UID: \"42570555-7956-4ee2-9c38-0c86484711ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb" Sep 10 23:46:40.070469 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:40.070321 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42570555-7956-4ee2-9c38-0c86484711ef-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb\" (UID: \"42570555-7956-4ee2-9c38-0c86484711ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb" Sep 10 23:46:40.070469 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:40.070356 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42570555-7956-4ee2-9c38-0c86484711ef-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb\" (UID: \"42570555-7956-4ee2-9c38-0c86484711ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb" Sep 10 23:46:40.070555 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:40.070479 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42570555-7956-4ee2-9c38-0c86484711ef-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb\" (UID: \"42570555-7956-4ee2-9c38-0c86484711ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb" Sep 10 23:46:40.070555 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:40.070516 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42570555-7956-4ee2-9c38-0c86484711ef-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb\" (UID: \"42570555-7956-4ee2-9c38-0c86484711ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb" Sep 10 23:46:40.070641 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:40.070576 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtxzn\" (UniqueName: \"kubernetes.io/projected/42570555-7956-4ee2-9c38-0c86484711ef-kube-api-access-mtxzn\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb\" (UID: \"42570555-7956-4ee2-9c38-0c86484711ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb" Sep 10 23:46:40.171105 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:40.171069 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42570555-7956-4ee2-9c38-0c86484711ef-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb\" (UID: \"42570555-7956-4ee2-9c38-0c86484711ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb" Sep 10 23:46:40.171105 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:40.171107 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42570555-7956-4ee2-9c38-0c86484711ef-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb\" (UID: \"42570555-7956-4ee2-9c38-0c86484711ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb" Sep 10 23:46:40.171355 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:40.171135 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtxzn\" (UniqueName: \"kubernetes.io/projected/42570555-7956-4ee2-9c38-0c86484711ef-kube-api-access-mtxzn\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb\" (UID: \"42570555-7956-4ee2-9c38-0c86484711ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb" Sep 10 23:46:40.171355 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:40.171264 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42570555-7956-4ee2-9c38-0c86484711ef-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb\" (UID: \"42570555-7956-4ee2-9c38-0c86484711ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb" Sep 10 23:46:40.171355 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:40.171312 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42570555-7956-4ee2-9c38-0c86484711ef-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb\" (UID: \"42570555-7956-4ee2-9c38-0c86484711ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb" Sep 10 23:46:40.171355 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:40.171352 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42570555-7956-4ee2-9c38-0c86484711ef-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb\" (UID: \"42570555-7956-4ee2-9c38-0c86484711ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb" Sep 10 23:46:40.171559 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:40.171522 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42570555-7956-4ee2-9c38-0c86484711ef-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb\" (UID: \"42570555-7956-4ee2-9c38-0c86484711ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb" Sep 10 23:46:40.171612 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:40.171568 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42570555-7956-4ee2-9c38-0c86484711ef-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb\" (UID: \"42570555-7956-4ee2-9c38-0c86484711ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb" Sep 10 23:46:40.171670 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:40.171653 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42570555-7956-4ee2-9c38-0c86484711ef-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb\" (UID: \"42570555-7956-4ee2-9c38-0c86484711ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb" Sep 10 23:46:40.173595 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:40.173568 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42570555-7956-4ee2-9c38-0c86484711ef-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb\" (UID: \"42570555-7956-4ee2-9c38-0c86484711ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb" Sep 10 23:46:40.173760 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:40.173742 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42570555-7956-4ee2-9c38-0c86484711ef-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb\" (UID: \"42570555-7956-4ee2-9c38-0c86484711ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb" Sep 10 23:46:40.178416 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:40.178391 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtxzn\" (UniqueName: \"kubernetes.io/projected/42570555-7956-4ee2-9c38-0c86484711ef-kube-api-access-mtxzn\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb\" (UID: \"42570555-7956-4ee2-9c38-0c86484711ef\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb" Sep 10 23:46:40.334700 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:40.334620 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb" Sep 10 23:46:40.603802 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:40.603776 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb"] Sep 10 23:46:40.606555 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:46:40.606528 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42570555_7956_4ee2_9c38_0c86484711ef.slice/crio-9b656ca7eec9bad4a713e5b34c751ee393da6c43a1db4a8bf589a255ba765358 WatchSource:0}: Error finding container 9b656ca7eec9bad4a713e5b34c751ee393da6c43a1db4a8bf589a255ba765358: Status 404 returned error can't find the container with id 9b656ca7eec9bad4a713e5b34c751ee393da6c43a1db4a8bf589a255ba765358 Sep 10 23:46:41.494065 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:41.494025 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb" event={"ID":"42570555-7956-4ee2-9c38-0c86484711ef","Type":"ContainerStarted","Data":"00cf004e833c1d6c854386a5db29688bbec30168dfdd741caaf4bacbb4f5bf38"} Sep 10 23:46:41.494548 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:41.494085 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb" event={"ID":"42570555-7956-4ee2-9c38-0c86484711ef","Type":"ContainerStarted","Data":"9b656ca7eec9bad4a713e5b34c751ee393da6c43a1db4a8bf589a255ba765358"} Sep 10 23:46:41.512153 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:41.512106 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb" podStartSLOduration=1.337192228 podStartE2EDuration="1.512090645s" totalImagesPullingTime="174.898417ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:46:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:46:41.511451606 +0000 UTC m=+616.978925932" watchObservedRunningTime="2026-09-10 23:46:41.512090645 +0000 UTC m=+616.979564970" Sep 10 23:46:50.334795 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:50.334697 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb" Sep 10 23:46:50.334795 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:50.334772 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb" Sep 10 23:46:50.347094 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:50.347070 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb" Sep 10 23:46:50.544091 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:46:50.544051 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb" Sep 10 23:47:07.515926 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:07.515890 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl"] Sep 10 23:47:07.519315 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:07.519285 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl" Sep 10 23:47:07.521883 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:07.521858 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 10 23:47:07.531334 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:07.531308 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl"] Sep 10 23:47:07.603447 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:07.603416 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl\" (UID: \"6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl" Sep 10 23:47:07.603677 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:07.603457 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl\" (UID: \"6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl" Sep 10 23:47:07.603677 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:07.603496 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl\" (UID: \"6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl" Sep 10 23:47:07.603677 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:07.603519 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl\" (UID: \"6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl" Sep 10 23:47:07.603677 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:07.603557 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frcsk\" (UniqueName: \"kubernetes.io/projected/6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab-kube-api-access-frcsk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl\" (UID: \"6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl" Sep 10 23:47:07.603677 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:07.603615 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl\" (UID: \"6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl" Sep 10 23:47:07.704450 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:07.704420 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl\" (UID: \"6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl" Sep 10 23:47:07.704450 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:07.704454 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl\" (UID: \"6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl" Sep 10 23:47:07.704682 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:07.704478 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl\" (UID: \"6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl" Sep 10 23:47:07.704682 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:07.704502 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl\" (UID: \"6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl" Sep 10 23:47:07.704682 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:07.704544 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frcsk\" (UniqueName: \"kubernetes.io/projected/6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab-kube-api-access-frcsk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl\" (UID: \"6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl" Sep 10 23:47:07.704682 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:07.704579 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl\" (UID: \"6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl" Sep 10 23:47:07.704994 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:07.704967 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl\" (UID: \"6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl" Sep 10 23:47:07.705097 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:07.705018 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl\" (UID: \"6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl" Sep 10 23:47:07.705097 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:07.705030 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl\" (UID: \"6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl" Sep 10 23:47:07.706808 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:07.706780 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl\" (UID: \"6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl" Sep 10 23:47:07.707138 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:07.707094 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl\" (UID: \"6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl" Sep 10 23:47:07.712650 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:07.712627 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frcsk\" (UniqueName: \"kubernetes.io/projected/6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab-kube-api-access-frcsk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl\" (UID: \"6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl" Sep 10 23:47:07.829550 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:07.829468 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl" Sep 10 23:47:08.099557 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:08.099527 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl"] Sep 10 23:47:08.102393 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:47:08.102363 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebaa8c0_7bb2_4bbc_b351_92c3cb89d3ab.slice/crio-514824a17d70dbff756b578725f536e5e18c136b0212fd89fa2af45769faaf5e WatchSource:0}: Error finding container 514824a17d70dbff756b578725f536e5e18c136b0212fd89fa2af45769faaf5e: Status 404 returned error can't find the container with id 514824a17d70dbff756b578725f536e5e18c136b0212fd89fa2af45769faaf5e Sep 10 23:47:08.104130 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:08.104112 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 23:47:08.612587 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:08.612547 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl" event={"ID":"6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab","Type":"ContainerStarted","Data":"224753246f9a7a00d37f15ff8d3f29af08f9f81a24a9a33ea05f7a19dcef425b"} Sep 10 23:47:08.612998 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:08.612613 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl" event={"ID":"6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab","Type":"ContainerStarted","Data":"514824a17d70dbff756b578725f536e5e18c136b0212fd89fa2af45769faaf5e"} Sep 10 23:47:08.630149 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:08.630086 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl" podStartSLOduration=1.468990913 podStartE2EDuration="1.630069633s" totalImagesPullingTime="161.07872ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:47:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:47:08.628284212 +0000 UTC m=+644.095758538" watchObservedRunningTime="2026-09-10 23:47:08.630069633 +0000 UTC m=+644.097543957" Sep 10 23:47:17.829583 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:17.829545 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl" Sep 10 23:47:17.830083 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:17.829612 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl" Sep 10 23:47:17.841741 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:17.841719 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl" Sep 10 23:47:18.665410 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:47:18.665385 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl" Sep 10 23:49:48.814861 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:48.814784 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d"] Sep 10 23:49:48.817915 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:48.817898 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:48.821562 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:48.821539 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ce1dd5-openshift-default-dockercfg-npcnh\"" Sep 10 23:49:48.834557 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:48.834530 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d"] Sep 10 23:49:48.914003 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:48.913973 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istio-envoy\") pod \"e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:48.914003 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:48.914007 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-credential-socket\") pod \"e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:48.914196 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:48.914026 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhr2s\" (UniqueName: \"kubernetes.io/projected/2a673735-8cb8-4342-81a1-e7ec4357fbf7-kube-api-access-xhr2s\") pod \"e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:48.914196 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:48.914073 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istio-token\") pod \"e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:48.914196 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:48.914099 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istiod-ca-cert\") pod \"e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:48.914196 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:48.914186 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istio-podinfo\") pod \"e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:48.914320 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:48.914222 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-workload-certs\") pod \"e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:48.914320 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:48.914267 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istio-data\") pod \"e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:48.914320 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:48.914304 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-workload-socket\") pod \"e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:49.015076 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:49.015042 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istio-envoy\") pod \"e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:49.015076 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:49.015077 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-credential-socket\") pod \"e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:49.015305 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:49.015093 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhr2s\" (UniqueName: \"kubernetes.io/projected/2a673735-8cb8-4342-81a1-e7ec4357fbf7-kube-api-access-xhr2s\") pod \"e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:49.015305 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:49.015117 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istio-token\") pod \"e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:49.015305 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:49.015150 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istiod-ca-cert\") pod \"e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:49.015305 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:49.015203 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istio-podinfo\") pod \"e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:49.015305 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:49.015238 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-workload-certs\") pod \"e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:49.015305 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:49.015260 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istio-data\") pod \"e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:49.015610 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:49.015313 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-workload-socket\") pod \"e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:49.015610 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:49.015562 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-credential-socket\") pod \"e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:49.015732 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:49.015707 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-workload-certs\") pod \"e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:49.015867 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:49.015842 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-workload-socket\") pod \"e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:49.015959 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:49.015893 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istio-data\") pod \"e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:49.016053 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:49.016033 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istiod-ca-cert\") pod \"e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:49.017388 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:49.017370 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istio-envoy\") pod \"e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:49.017727 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:49.017711 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istio-podinfo\") pod \"e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:49.023331 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:49.023302 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhr2s\" (UniqueName: \"kubernetes.io/projected/2a673735-8cb8-4342-81a1-e7ec4357fbf7-kube-api-access-xhr2s\") pod \"e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:49.023433 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:49.023414 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istio-token\") pod \"e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:49.128434 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:49.128410 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:49.406261 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:49.406184 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d"] Sep 10 23:49:49.409149 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:49:49.409125 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a673735_8cb8_4342_81a1_e7ec4357fbf7.slice/crio-54a8114335ef5ca6584edc0718ef61cec98d74956966713dc598cfa203f2c45a WatchSource:0}: Error finding container 54a8114335ef5ca6584edc0718ef61cec98d74956966713dc598cfa203f2c45a: Status 404 returned error can't find the container with id 54a8114335ef5ca6584edc0718ef61cec98d74956966713dc598cfa203f2c45a Sep 10 23:49:49.411219 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:49.411191 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:49:49.411302 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:49.411256 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:49:49.411302 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:49.411286 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:49:50.288352 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:50.288314 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" event={"ID":"2a673735-8cb8-4342-81a1-e7ec4357fbf7","Type":"ContainerStarted","Data":"bacc0c57698050fc456b6ccc247511aa2bcf43c84c780d822a9a68302d7f01fa"} Sep 10 23:49:50.288811 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:50.288382 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" event={"ID":"2a673735-8cb8-4342-81a1-e7ec4357fbf7","Type":"ContainerStarted","Data":"54a8114335ef5ca6584edc0718ef61cec98d74956966713dc598cfa203f2c45a"} Sep 10 23:49:50.310912 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:50.310861 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" podStartSLOduration=2.310819911 podStartE2EDuration="2.310819911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:49:50.308988622 +0000 UTC m=+805.776462948" watchObservedRunningTime="2026-09-10 23:49:50.310819911 +0000 UTC m=+805.778294236" Sep 10 23:49:51.128744 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:51.128709 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:51.133212 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:51.133185 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:51.293481 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:51.293450 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:51.294505 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:51.294484 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:49:54.698900 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:54.698864 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm"] Sep 10 23:49:54.702226 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:54.702202 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm" Sep 10 23:49:54.708033 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:54.708014 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ce1dd5-dockercfg-rnv8t\"" Sep 10 23:49:54.708300 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:54.708281 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ce1dd5\"" Sep 10 23:49:54.715960 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:54.715915 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm"] Sep 10 23:49:54.757537 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:54.757499 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25"] Sep 10 23:49:54.760690 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:54.760670 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25" Sep 10 23:49:54.763110 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:54.763089 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff709b92-c05f-41b6-8a5c-e0f59549ecd7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm\" (UID: \"ff709b92-c05f-41b6-8a5c-e0f59549ecd7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm" Sep 10 23:49:54.763203 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:54.763129 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4smvj\" (UniqueName: \"kubernetes.io/projected/ff709b92-c05f-41b6-8a5c-e0f59549ecd7-kube-api-access-4smvj\") pod \"payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm\" (UID: \"ff709b92-c05f-41b6-8a5c-e0f59549ecd7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm" Sep 10 23:49:54.782127 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:54.782099 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25"] Sep 10 23:49:54.863846 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:54.863813 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff709b92-c05f-41b6-8a5c-e0f59549ecd7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm\" (UID: \"ff709b92-c05f-41b6-8a5c-e0f59549ecd7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm" Sep 10 23:49:54.864048 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:54.863860 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4bx5\" (UniqueName: \"kubernetes.io/projected/f42dfd8b-5bb6-4121-994a-0a5ff684470a-kube-api-access-t4bx5\") pod \"payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25\" (UID: \"f42dfd8b-5bb6-4121-994a-0a5ff684470a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25" Sep 10 23:49:54.864048 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:54.863885 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4smvj\" (UniqueName: \"kubernetes.io/projected/ff709b92-c05f-41b6-8a5c-e0f59549ecd7-kube-api-access-4smvj\") pod \"payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm\" (UID: \"ff709b92-c05f-41b6-8a5c-e0f59549ecd7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm" Sep 10 23:49:54.864048 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:54.863917 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f42dfd8b-5bb6-4121-994a-0a5ff684470a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25\" (UID: \"f42dfd8b-5bb6-4121-994a-0a5ff684470a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25" Sep 10 23:49:54.864567 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:54.864547 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff709b92-c05f-41b6-8a5c-e0f59549ecd7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm\" (UID: \"ff709b92-c05f-41b6-8a5c-e0f59549ecd7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm" Sep 10 23:49:54.873025 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:54.873002 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4smvj\" (UniqueName: \"kubernetes.io/projected/ff709b92-c05f-41b6-8a5c-e0f59549ecd7-kube-api-access-4smvj\") pod \"payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm\" (UID: \"ff709b92-c05f-41b6-8a5c-e0f59549ecd7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm" Sep 10 23:49:54.964633 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:54.964551 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4bx5\" (UniqueName: \"kubernetes.io/projected/f42dfd8b-5bb6-4121-994a-0a5ff684470a-kube-api-access-t4bx5\") pod \"payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25\" (UID: \"f42dfd8b-5bb6-4121-994a-0a5ff684470a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25" Sep 10 23:49:54.964633 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:54.964612 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f42dfd8b-5bb6-4121-994a-0a5ff684470a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25\" (UID: \"f42dfd8b-5bb6-4121-994a-0a5ff684470a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25" Sep 10 23:49:54.965380 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:54.965355 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f42dfd8b-5bb6-4121-994a-0a5ff684470a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25\" (UID: \"f42dfd8b-5bb6-4121-994a-0a5ff684470a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25" Sep 10 23:49:54.972778 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:54.972757 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4bx5\" (UniqueName: \"kubernetes.io/projected/f42dfd8b-5bb6-4121-994a-0a5ff684470a-kube-api-access-t4bx5\") pod \"payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25\" (UID: \"f42dfd8b-5bb6-4121-994a-0a5ff684470a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25" Sep 10 23:49:55.012604 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:55.012584 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm" Sep 10 23:49:55.069911 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:55.069866 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25" Sep 10 23:49:55.290008 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:55.289972 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm"] Sep 10 23:49:55.292558 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:49:55.292532 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff709b92_c05f_41b6_8a5c_e0f59549ecd7.slice/crio-8d706c126db39e3ce1e439c4089b4294b1538b0f204a0aaa0cc3eb98184cc0d6 WatchSource:0}: Error finding container 8d706c126db39e3ce1e439c4089b4294b1538b0f204a0aaa0cc3eb98184cc0d6: Status 404 returned error can't find the container with id 8d706c126db39e3ce1e439c4089b4294b1538b0f204a0aaa0cc3eb98184cc0d6 Sep 10 23:49:55.313249 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:55.313225 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm" event={"ID":"ff709b92-c05f-41b6-8a5c-e0f59549ecd7","Type":"ContainerStarted","Data":"8d706c126db39e3ce1e439c4089b4294b1538b0f204a0aaa0cc3eb98184cc0d6"} Sep 10 23:49:55.353112 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:55.353086 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25"] Sep 10 23:49:55.355923 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:49:55.355895 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf42dfd8b_5bb6_4121_994a_0a5ff684470a.slice/crio-276934f1dc69cd08af63257962e6fde8ba2951807a7719a4a561c14c0a2d638a WatchSource:0}: Error finding container 276934f1dc69cd08af63257962e6fde8ba2951807a7719a4a561c14c0a2d638a: Status 404 returned error can't find the container with id 276934f1dc69cd08af63257962e6fde8ba2951807a7719a4a561c14c0a2d638a Sep 10 23:49:56.319513 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:56.319481 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25" event={"ID":"f42dfd8b-5bb6-4121-994a-0a5ff684470a","Type":"ContainerStarted","Data":"939f8b82d75a1b557b97fa4217e663ad38521bf50350fa3f4203cd1d8838d22a"} Sep 10 23:49:56.319928 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:56.319530 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25" event={"ID":"f42dfd8b-5bb6-4121-994a-0a5ff684470a","Type":"ContainerStarted","Data":"276934f1dc69cd08af63257962e6fde8ba2951807a7719a4a561c14c0a2d638a"} Sep 10 23:49:56.319928 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:56.319569 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25" Sep 10 23:49:56.320997 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:56.320975 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm" event={"ID":"ff709b92-c05f-41b6-8a5c-e0f59549ecd7","Type":"ContainerStarted","Data":"d92444167ebf41f2f3a2dd2b464b7359d0ee281cf26fc38ae91e29ee54682590"} Sep 10 23:49:56.321105 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:56.321087 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm" Sep 10 23:49:56.337175 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:56.337129 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25" podStartSLOduration=2.337113112 podStartE2EDuration="2.337113112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:49:56.336238501 +0000 UTC m=+811.803712827" watchObservedRunningTime="2026-09-10 23:49:56.337113112 +0000 UTC m=+811.804587438" Sep 10 23:49:56.360954 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:49:56.360122 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm" podStartSLOduration=2.360101946 podStartE2EDuration="2.360101946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:49:56.357034586 +0000 UTC m=+811.824508913" watchObservedRunningTime="2026-09-10 23:49:56.360101946 +0000 UTC m=+811.827576271" Sep 10 23:50:02.186732 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:02.186695 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh"] Sep 10 23:50:02.190312 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:02.190291 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh" Sep 10 23:50:02.192733 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:02.192710 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-643c297d\"" Sep 10 23:50:02.193181 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:02.193158 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-643c297d-dockercfg-sd459\"" Sep 10 23:50:02.201240 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:02.201219 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-643c297d-65cb59db7-2l894"] Sep 10 23:50:02.206591 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:02.206570 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh"] Sep 10 23:50:02.206708 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:02.206689 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-643c297d-65cb59db7-2l894" Sep 10 23:50:02.217923 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:02.217898 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-643c297d-65cb59db7-2l894"] Sep 10 23:50:02.327195 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:02.327162 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4dce57-35c7-49a7-886f-25cddab4c0d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh\" (UID: \"1c4dce57-35c7-49a7-886f-25cddab4c0d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh" Sep 10 23:50:02.327341 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:02.327209 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqbx6\" (UniqueName: \"kubernetes.io/projected/9ad27453-b89c-41db-9111-e84973fc68e0-kube-api-access-gqbx6\") pod \"payload-processing-e2e-ait-643c297d-65cb59db7-2l894\" (UID: \"9ad27453-b89c-41db-9111-e84973fc68e0\") " pod="openshift-ingress/payload-processing-e2e-ait-643c297d-65cb59db7-2l894" Sep 10 23:50:02.327377 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:02.327332 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad27453-b89c-41db-9111-e84973fc68e0-plugins-config-volume\") pod \"payload-processing-e2e-ait-643c297d-65cb59db7-2l894\" (UID: \"9ad27453-b89c-41db-9111-e84973fc68e0\") " pod="openshift-ingress/payload-processing-e2e-ait-643c297d-65cb59db7-2l894" Sep 10 23:50:02.327377 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:02.327366 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffxgs\" (UniqueName: \"kubernetes.io/projected/1c4dce57-35c7-49a7-886f-25cddab4c0d6-kube-api-access-ffxgs\") pod \"payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh\" (UID: \"1c4dce57-35c7-49a7-886f-25cddab4c0d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh" Sep 10 23:50:02.428202 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:02.428166 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4dce57-35c7-49a7-886f-25cddab4c0d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh\" (UID: \"1c4dce57-35c7-49a7-886f-25cddab4c0d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh" Sep 10 23:50:02.428363 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:02.428219 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqbx6\" (UniqueName: \"kubernetes.io/projected/9ad27453-b89c-41db-9111-e84973fc68e0-kube-api-access-gqbx6\") pod \"payload-processing-e2e-ait-643c297d-65cb59db7-2l894\" (UID: \"9ad27453-b89c-41db-9111-e84973fc68e0\") " pod="openshift-ingress/payload-processing-e2e-ait-643c297d-65cb59db7-2l894" Sep 10 23:50:02.428363 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:02.428312 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad27453-b89c-41db-9111-e84973fc68e0-plugins-config-volume\") pod \"payload-processing-e2e-ait-643c297d-65cb59db7-2l894\" (UID: \"9ad27453-b89c-41db-9111-e84973fc68e0\") " pod="openshift-ingress/payload-processing-e2e-ait-643c297d-65cb59db7-2l894" Sep 10 23:50:02.428363 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:02.428345 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffxgs\" (UniqueName: \"kubernetes.io/projected/1c4dce57-35c7-49a7-886f-25cddab4c0d6-kube-api-access-ffxgs\") pod \"payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh\" (UID: \"1c4dce57-35c7-49a7-886f-25cddab4c0d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh" Sep 10 23:50:02.428931 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:02.428910 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4dce57-35c7-49a7-886f-25cddab4c0d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh\" (UID: \"1c4dce57-35c7-49a7-886f-25cddab4c0d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh" Sep 10 23:50:02.429049 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:02.428996 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad27453-b89c-41db-9111-e84973fc68e0-plugins-config-volume\") pod \"payload-processing-e2e-ait-643c297d-65cb59db7-2l894\" (UID: \"9ad27453-b89c-41db-9111-e84973fc68e0\") " pod="openshift-ingress/payload-processing-e2e-ait-643c297d-65cb59db7-2l894" Sep 10 23:50:02.453637 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:02.453559 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqbx6\" (UniqueName: \"kubernetes.io/projected/9ad27453-b89c-41db-9111-e84973fc68e0-kube-api-access-gqbx6\") pod \"payload-processing-e2e-ait-643c297d-65cb59db7-2l894\" (UID: \"9ad27453-b89c-41db-9111-e84973fc68e0\") " pod="openshift-ingress/payload-processing-e2e-ait-643c297d-65cb59db7-2l894" Sep 10 23:50:02.453637 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:02.453612 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffxgs\" (UniqueName: \"kubernetes.io/projected/1c4dce57-35c7-49a7-886f-25cddab4c0d6-kube-api-access-ffxgs\") pod \"payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh\" (UID: \"1c4dce57-35c7-49a7-886f-25cddab4c0d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh" Sep 10 23:50:02.505045 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:02.505012 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh" Sep 10 23:50:02.520202 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:02.520174 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-643c297d-65cb59db7-2l894" Sep 10 23:50:02.805654 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:02.805622 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh"] Sep 10 23:50:02.806349 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:50:02.806323 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4dce57_35c7_49a7_886f_25cddab4c0d6.slice/crio-4c9db912b6f547e838fb48eb541557138a32d34bc642ef0e12674010263843d3 WatchSource:0}: Error finding container 4c9db912b6f547e838fb48eb541557138a32d34bc642ef0e12674010263843d3: Status 404 returned error can't find the container with id 4c9db912b6f547e838fb48eb541557138a32d34bc642ef0e12674010263843d3 Sep 10 23:50:02.839968 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:02.838356 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-643c297d-65cb59db7-2l894"] Sep 10 23:50:03.365539 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:03.364611 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh" event={"ID":"1c4dce57-35c7-49a7-886f-25cddab4c0d6","Type":"ContainerStarted","Data":"6d4be29f3571c1fe52eaffb2846bd42fa3e8742f15fae224f607fee6be817e4d"} Sep 10 23:50:03.365539 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:03.364682 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh" event={"ID":"1c4dce57-35c7-49a7-886f-25cddab4c0d6","Type":"ContainerStarted","Data":"4c9db912b6f547e838fb48eb541557138a32d34bc642ef0e12674010263843d3"} Sep 10 23:50:03.365539 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:03.365478 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh" Sep 10 23:50:03.367861 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:03.367831 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-643c297d-65cb59db7-2l894" event={"ID":"9ad27453-b89c-41db-9111-e84973fc68e0","Type":"ContainerStarted","Data":"ee29db4a74d502c14b82d479bcce6c9c43ff9135dfa2c89be36fc626941a0381"} Sep 10 23:50:03.368001 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:03.367884 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-643c297d-65cb59db7-2l894" event={"ID":"9ad27453-b89c-41db-9111-e84973fc68e0","Type":"ContainerStarted","Data":"b481c3713b10b4b70960190eefa7b63b82963ddefc6d204094276145cc196022"} Sep 10 23:50:03.368001 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:03.367968 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-643c297d-65cb59db7-2l894" Sep 10 23:50:03.398319 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:03.398266 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh" podStartSLOduration=1.398248933 podStartE2EDuration="1.398248933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:03.397097168 +0000 UTC m=+818.864571504" watchObservedRunningTime="2026-09-10 23:50:03.398248933 +0000 UTC m=+818.865723258" Sep 10 23:50:03.416244 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:03.416200 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-643c297d-65cb59db7-2l894" podStartSLOduration=1.416185217 podStartE2EDuration="1.416185217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:03.414504325 +0000 UTC m=+818.881978661" watchObservedRunningTime="2026-09-10 23:50:03.416185217 +0000 UTC m=+818.883659541" Sep 10 23:50:05.858054 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:05.858014 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg"] Sep 10 23:50:05.862096 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:05.862070 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg" Sep 10 23:50:05.864918 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:05.864894 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-66836f-dockercfg-gv2bn\"" Sep 10 23:50:05.865142 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:05.864999 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-66836f\"" Sep 10 23:50:05.875536 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:05.875511 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg"] Sep 10 23:50:05.973524 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:05.973482 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqkpx\" (UniqueName: \"kubernetes.io/projected/5aab5922-dba3-4f74-a65d-63b61dc1e138-kube-api-access-hqkpx\") pod \"payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg\" (UID: \"5aab5922-dba3-4f74-a65d-63b61dc1e138\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg" Sep 10 23:50:05.973687 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:05.973618 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aab5922-dba3-4f74-a65d-63b61dc1e138-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg\" (UID: \"5aab5922-dba3-4f74-a65d-63b61dc1e138\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg" Sep 10 23:50:06.074355 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:06.074311 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aab5922-dba3-4f74-a65d-63b61dc1e138-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg\" (UID: \"5aab5922-dba3-4f74-a65d-63b61dc1e138\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg" Sep 10 23:50:06.074501 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:06.074427 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqkpx\" (UniqueName: \"kubernetes.io/projected/5aab5922-dba3-4f74-a65d-63b61dc1e138-kube-api-access-hqkpx\") pod \"payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg\" (UID: \"5aab5922-dba3-4f74-a65d-63b61dc1e138\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg" Sep 10 23:50:06.075049 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:06.075029 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aab5922-dba3-4f74-a65d-63b61dc1e138-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg\" (UID: \"5aab5922-dba3-4f74-a65d-63b61dc1e138\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg" Sep 10 23:50:06.084587 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:06.084558 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqkpx\" (UniqueName: \"kubernetes.io/projected/5aab5922-dba3-4f74-a65d-63b61dc1e138-kube-api-access-hqkpx\") pod \"payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg\" (UID: \"5aab5922-dba3-4f74-a65d-63b61dc1e138\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg" Sep 10 23:50:06.174659 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:06.174627 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg" Sep 10 23:50:06.451548 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:06.451480 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg"] Sep 10 23:50:07.327126 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:07.327097 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25" Sep 10 23:50:07.327560 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:07.327340 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm" Sep 10 23:50:07.388559 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:07.388523 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg" event={"ID":"5aab5922-dba3-4f74-a65d-63b61dc1e138","Type":"ContainerStarted","Data":"29ab92a12fad5767861f4b78705a9a05af1ed40e0ebecfb298b03ec45a8e6cc7"} Sep 10 23:50:07.388750 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:07.388584 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg" event={"ID":"5aab5922-dba3-4f74-a65d-63b61dc1e138","Type":"ContainerStarted","Data":"728808ab4d87633d29ad5206a62308d0b808f052b0b3828262e8c493820c0872"} Sep 10 23:50:07.388750 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:07.388674 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg" Sep 10 23:50:07.408834 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:07.408780 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg" podStartSLOduration=2.408757182 podStartE2EDuration="2.408757182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:07.407818482 +0000 UTC m=+822.875292834" watchObservedRunningTime="2026-09-10 23:50:07.408757182 +0000 UTC m=+822.876231537" Sep 10 23:50:10.403356 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:10.403321 2578 generic.go:356] "Generic (PLEG): container finished" podID="2a673735-8cb8-4342-81a1-e7ec4357fbf7" containerID="bacc0c57698050fc456b6ccc247511aa2bcf43c84c780d822a9a68302d7f01fa" exitCode=0 Sep 10 23:50:10.403761 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:10.403396 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" event={"ID":"2a673735-8cb8-4342-81a1-e7ec4357fbf7","Type":"ContainerDied","Data":"bacc0c57698050fc456b6ccc247511aa2bcf43c84c780d822a9a68302d7f01fa"} Sep 10 23:50:10.403851 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:10.403835 2578 scope.go:117] "RemoveContainer" containerID="bacc0c57698050fc456b6ccc247511aa2bcf43c84c780d822a9a68302d7f01fa" Sep 10 23:50:10.404879 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:10.404854 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:50:10.404929 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:10.404916 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:50:10.404980 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:10.404963 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:50:11.410997 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:11.410930 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" event={"ID":"2a673735-8cb8-4342-81a1-e7ec4357fbf7","Type":"ContainerStarted","Data":"fd894e883ad79133f6d2e7a4ed6f1cf1cfdcceeb4274c7cf6a6b27c090ec1e74"} Sep 10 23:50:12.128805 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:12.128770 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:50:12.129016 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:12.128993 2578 patch_prober.go:28] interesting pod/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body= Sep 10 23:50:12.129066 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:12.129043 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" podUID="2a673735-8cb8-4342-81a1-e7ec4357fbf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" Sep 10 23:50:13.128780 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:13.128745 2578 patch_prober.go:28] interesting pod/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body= Sep 10 23:50:13.129201 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:13.128796 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" podUID="2a673735-8cb8-4342-81a1-e7ec4357fbf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" Sep 10 23:50:14.129118 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:14.129083 2578 patch_prober.go:28] interesting pod/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body= Sep 10 23:50:14.129490 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:14.129152 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" podUID="2a673735-8cb8-4342-81a1-e7ec4357fbf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" Sep 10 23:50:14.374120 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:14.374089 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-643c297d-65cb59db7-2l894" Sep 10 23:50:15.129109 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:15.129073 2578 patch_prober.go:28] interesting pod/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body= Sep 10 23:50:15.129278 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:15.129127 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" podUID="2a673735-8cb8-4342-81a1-e7ec4357fbf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" Sep 10 23:50:15.379094 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:15.379058 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh" Sep 10 23:50:16.129666 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:16.129629 2578 patch_prober.go:28] interesting pod/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body= Sep 10 23:50:16.130168 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:16.129692 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" podUID="2a673735-8cb8-4342-81a1-e7ec4357fbf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" Sep 10 23:50:17.236004 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:17.235970 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:50:17.236458 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:17.236436 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:50:17.237147 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:17.237132 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:50:18.394713 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:18.394687 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg" Sep 10 23:50:25.672378 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:25.672341 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643c297d-bq9xh"] Sep 10 23:50:25.675757 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:25.675739 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643c297d-bq9xh" Sep 10 23:50:25.678161 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:25.678137 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-22lrj\"" Sep 10 23:50:25.678277 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:25.678141 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 23:50:25.683669 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:25.683644 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643c297d-bq9xh"] Sep 10 23:50:25.753662 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:25.753634 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e4038b-17bf-41c1-9b9a-7cf91ac24da8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-643c297d-bq9xh\" (UID: \"47e4038b-17bf-41c1-9b9a-7cf91ac24da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643c297d-bq9xh" Sep 10 23:50:25.854357 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:25.854322 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e4038b-17bf-41c1-9b9a-7cf91ac24da8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-643c297d-bq9xh\" (UID: \"47e4038b-17bf-41c1-9b9a-7cf91ac24da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643c297d-bq9xh" Sep 10 23:50:25.855006 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:25.854985 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e4038b-17bf-41c1-9b9a-7cf91ac24da8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-643c297d-bq9xh\" (UID: \"47e4038b-17bf-41c1-9b9a-7cf91ac24da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643c297d-bq9xh" Sep 10 23:50:25.986201 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:25.986130 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643c297d-bq9xh" Sep 10 23:50:26.267112 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:26.267088 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643c297d-bq9xh"] Sep 10 23:50:26.268741 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:50:26.268713 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e4038b_17bf_41c1_9b9a_7cf91ac24da8.slice/crio-a59769bffe273a3b9d653b0c403eb825f0619dd8193b0acb84957d0bcb64855b WatchSource:0}: Error finding container a59769bffe273a3b9d653b0c403eb825f0619dd8193b0acb84957d0bcb64855b: Status 404 returned error can't find the container with id a59769bffe273a3b9d653b0c403eb825f0619dd8193b0acb84957d0bcb64855b Sep 10 23:50:26.484606 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:26.484572 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643c297d-bq9xh" event={"ID":"47e4038b-17bf-41c1-9b9a-7cf91ac24da8","Type":"ContainerStarted","Data":"a59769bffe273a3b9d653b0c403eb825f0619dd8193b0acb84957d0bcb64855b"} Sep 10 23:50:29.416820 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:29.416783 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg"] Sep 10 23:50:29.420261 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:29.420244 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg" Sep 10 23:50:29.422593 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:29.422563 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-13374d\"" Sep 10 23:50:29.422716 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:29.422698 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-13374d-dockercfg-l6b99\"" Sep 10 23:50:29.431044 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:29.431010 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg"] Sep 10 23:50:29.443578 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:29.443555 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj"] Sep 10 23:50:29.446857 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:29.446839 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj" Sep 10 23:50:29.458184 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:29.458152 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj"] Sep 10 23:50:29.487245 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:29.487220 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4tdm\" (UniqueName: \"kubernetes.io/projected/ea312b45-cf18-4c8d-a7e6-10a5ef475ee4-kube-api-access-x4tdm\") pod \"payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg\" (UID: \"ea312b45-cf18-4c8d-a7e6-10a5ef475ee4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg" Sep 10 23:50:29.487403 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:29.487262 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97npn\" (UniqueName: \"kubernetes.io/projected/ecbe850b-7c2f-4556-b6b5-f1aa2ec62703-kube-api-access-97npn\") pod \"payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj\" (UID: \"ecbe850b-7c2f-4556-b6b5-f1aa2ec62703\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj" Sep 10 23:50:29.487403 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:29.487293 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbe850b-7c2f-4556-b6b5-f1aa2ec62703-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj\" (UID: \"ecbe850b-7c2f-4556-b6b5-f1aa2ec62703\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj" Sep 10 23:50:29.487403 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:29.487383 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea312b45-cf18-4c8d-a7e6-10a5ef475ee4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg\" (UID: \"ea312b45-cf18-4c8d-a7e6-10a5ef475ee4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg" Sep 10 23:50:29.502295 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:29.502266 2578 generic.go:356] "Generic (PLEG): container finished" podID="47e4038b-17bf-41c1-9b9a-7cf91ac24da8" containerID="df254a9beccd3bf25e1ee41fe7facab558cc0d249fd5cb0e71f9adb336627902" exitCode=0 Sep 10 23:50:29.502295 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:29.502295 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643c297d-bq9xh" event={"ID":"47e4038b-17bf-41c1-9b9a-7cf91ac24da8","Type":"ContainerDied","Data":"df254a9beccd3bf25e1ee41fe7facab558cc0d249fd5cb0e71f9adb336627902"} Sep 10 23:50:29.588604 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:29.588566 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea312b45-cf18-4c8d-a7e6-10a5ef475ee4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg\" (UID: \"ea312b45-cf18-4c8d-a7e6-10a5ef475ee4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg" Sep 10 23:50:29.588772 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:29.588612 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4tdm\" (UniqueName: \"kubernetes.io/projected/ea312b45-cf18-4c8d-a7e6-10a5ef475ee4-kube-api-access-x4tdm\") pod \"payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg\" (UID: \"ea312b45-cf18-4c8d-a7e6-10a5ef475ee4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg" Sep 10 23:50:29.588772 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:29.588734 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97npn\" (UniqueName: \"kubernetes.io/projected/ecbe850b-7c2f-4556-b6b5-f1aa2ec62703-kube-api-access-97npn\") pod \"payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj\" (UID: \"ecbe850b-7c2f-4556-b6b5-f1aa2ec62703\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj" Sep 10 23:50:29.588855 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:29.588786 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbe850b-7c2f-4556-b6b5-f1aa2ec62703-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj\" (UID: \"ecbe850b-7c2f-4556-b6b5-f1aa2ec62703\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj" Sep 10 23:50:29.589283 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:29.589266 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea312b45-cf18-4c8d-a7e6-10a5ef475ee4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg\" (UID: \"ea312b45-cf18-4c8d-a7e6-10a5ef475ee4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg" Sep 10 23:50:29.589440 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:29.589424 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbe850b-7c2f-4556-b6b5-f1aa2ec62703-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj\" (UID: \"ecbe850b-7c2f-4556-b6b5-f1aa2ec62703\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj" Sep 10 23:50:29.596452 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:29.596422 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4tdm\" (UniqueName: \"kubernetes.io/projected/ea312b45-cf18-4c8d-a7e6-10a5ef475ee4-kube-api-access-x4tdm\") pod \"payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg\" (UID: \"ea312b45-cf18-4c8d-a7e6-10a5ef475ee4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg" Sep 10 23:50:29.596565 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:29.596424 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97npn\" (UniqueName: \"kubernetes.io/projected/ecbe850b-7c2f-4556-b6b5-f1aa2ec62703-kube-api-access-97npn\") pod \"payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj\" (UID: \"ecbe850b-7c2f-4556-b6b5-f1aa2ec62703\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj" Sep 10 23:50:29.730484 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:29.730403 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg" Sep 10 23:50:29.757769 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:29.757738 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj" Sep 10 23:50:30.009697 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:30.009667 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg"] Sep 10 23:50:30.011492 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:50:30.011464 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea312b45_cf18_4c8d_a7e6_10a5ef475ee4.slice/crio-93ece356570d1094ce8022722530989c383196628be5c1deb1ce16161401ff0f WatchSource:0}: Error finding container 93ece356570d1094ce8022722530989c383196628be5c1deb1ce16161401ff0f: Status 404 returned error can't find the container with id 93ece356570d1094ce8022722530989c383196628be5c1deb1ce16161401ff0f Sep 10 23:50:30.043275 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:30.043243 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj"] Sep 10 23:50:30.046092 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:50:30.046065 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecbe850b_7c2f_4556_b6b5_f1aa2ec62703.slice/crio-6dc87fb689e49dc9cdd93c4d86b971f7489426e04f7dc2b4f3f04be2461278b2 WatchSource:0}: Error finding container 6dc87fb689e49dc9cdd93c4d86b971f7489426e04f7dc2b4f3f04be2461278b2: Status 404 returned error can't find the container with id 6dc87fb689e49dc9cdd93c4d86b971f7489426e04f7dc2b4f3f04be2461278b2 Sep 10 23:50:30.509134 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:30.509099 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj" event={"ID":"ecbe850b-7c2f-4556-b6b5-f1aa2ec62703","Type":"ContainerStarted","Data":"c4ff20d27dd49e6ffe501def82008d2d03e8852a15cba0346102e17345016914"} Sep 10 23:50:30.509591 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:30.509159 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj" event={"ID":"ecbe850b-7c2f-4556-b6b5-f1aa2ec62703","Type":"ContainerStarted","Data":"6dc87fb689e49dc9cdd93c4d86b971f7489426e04f7dc2b4f3f04be2461278b2"} Sep 10 23:50:30.509591 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:30.509188 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj" Sep 10 23:50:30.510679 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:30.510655 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg" event={"ID":"ea312b45-cf18-4c8d-a7e6-10a5ef475ee4","Type":"ContainerStarted","Data":"061628c7fccfc66253f14ac07924e9f9ac0068e364e644da53c5e429531afe63"} Sep 10 23:50:30.510777 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:30.510701 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg" event={"ID":"ea312b45-cf18-4c8d-a7e6-10a5ef475ee4","Type":"ContainerStarted","Data":"93ece356570d1094ce8022722530989c383196628be5c1deb1ce16161401ff0f"} Sep 10 23:50:30.510857 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:30.510840 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg" Sep 10 23:50:30.552843 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:30.552762 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj" podStartSLOduration=1.55273968 podStartE2EDuration="1.55273968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:30.550727156 +0000 UTC m=+846.018201483" watchObservedRunningTime="2026-09-10 23:50:30.55273968 +0000 UTC m=+846.020214024" Sep 10 23:50:30.588587 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:30.588530 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg" podStartSLOduration=1.5885092859999999 podStartE2EDuration="1.588509286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:30.588238469 +0000 UTC m=+846.055712795" watchObservedRunningTime="2026-09-10 23:50:30.588509286 +0000 UTC m=+846.055983616" Sep 10 23:50:30.653904 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:30.653873 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643c297d-bq9xh" Sep 10 23:50:30.699003 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:30.698962 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e4038b-17bf-41c1-9b9a-7cf91ac24da8-maas-api-service-ca\") pod \"47e4038b-17bf-41c1-9b9a-7cf91ac24da8\" (UID: \"47e4038b-17bf-41c1-9b9a-7cf91ac24da8\") " Sep 10 23:50:30.699287 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:30.699262 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e4038b-17bf-41c1-9b9a-7cf91ac24da8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "47e4038b-17bf-41c1-9b9a-7cf91ac24da8" (UID: "47e4038b-17bf-41c1-9b9a-7cf91ac24da8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:50:30.806390 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:30.800722 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e4038b-17bf-41c1-9b9a-7cf91ac24da8-maas-api-service-ca\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:50:31.516758 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:31.516721 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643c297d-bq9xh" Sep 10 23:50:31.517293 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:31.516756 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643c297d-bq9xh" event={"ID":"47e4038b-17bf-41c1-9b9a-7cf91ac24da8","Type":"ContainerDied","Data":"a59769bffe273a3b9d653b0c403eb825f0619dd8193b0acb84957d0bcb64855b"} Sep 10 23:50:31.517293 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:31.516811 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a59769bffe273a3b9d653b0c403eb825f0619dd8193b0acb84957d0bcb64855b" Sep 10 23:50:41.518888 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:41.518850 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj" Sep 10 23:50:41.519387 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:41.519321 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg" Sep 10 23:50:43.089093 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.089057 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-643c297d-65cb59db7-2l894"] Sep 10 23:50:43.089504 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.089332 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-643c297d-65cb59db7-2l894" podUID="9ad27453-b89c-41db-9111-e84973fc68e0" containerName="payload-processing" containerID="cri-o://ee29db4a74d502c14b82d479bcce6c9c43ff9135dfa2c89be36fc626941a0381" gracePeriod=30 Sep 10 23:50:43.104606 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.104577 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh"] Sep 10 23:50:43.104863 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.104812 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh" podUID="1c4dce57-35c7-49a7-886f-25cddab4c0d6" containerName="payload-pre-processing" containerID="cri-o://6d4be29f3571c1fe52eaffb2846bd42fa3e8742f15fae224f607fee6be817e4d" gracePeriod=30 Sep 10 23:50:43.379858 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.379830 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-643c297d-65cb59db7-2l894" Sep 10 23:50:43.382768 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.382745 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh" Sep 10 23:50:43.452718 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.452685 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643c297d-bq9xh"] Sep 10 23:50:43.455477 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.455453 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-643c297d-bq9xh"] Sep 10 23:50:43.521162 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.521128 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffxgs\" (UniqueName: \"kubernetes.io/projected/1c4dce57-35c7-49a7-886f-25cddab4c0d6-kube-api-access-ffxgs\") pod \"1c4dce57-35c7-49a7-886f-25cddab4c0d6\" (UID: \"1c4dce57-35c7-49a7-886f-25cddab4c0d6\") " Sep 10 23:50:43.521343 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.521189 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad27453-b89c-41db-9111-e84973fc68e0-plugins-config-volume\") pod \"9ad27453-b89c-41db-9111-e84973fc68e0\" (UID: \"9ad27453-b89c-41db-9111-e84973fc68e0\") " Sep 10 23:50:43.521343 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.521256 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4dce57-35c7-49a7-886f-25cddab4c0d6-plugins-config-volume\") pod \"1c4dce57-35c7-49a7-886f-25cddab4c0d6\" (UID: \"1c4dce57-35c7-49a7-886f-25cddab4c0d6\") " Sep 10 23:50:43.521430 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.521383 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqbx6\" (UniqueName: \"kubernetes.io/projected/9ad27453-b89c-41db-9111-e84973fc68e0-kube-api-access-gqbx6\") pod \"9ad27453-b89c-41db-9111-e84973fc68e0\" (UID: \"9ad27453-b89c-41db-9111-e84973fc68e0\") " Sep 10 23:50:43.521682 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.521647 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c4dce57-35c7-49a7-886f-25cddab4c0d6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1c4dce57-35c7-49a7-886f-25cddab4c0d6" (UID: "1c4dce57-35c7-49a7-886f-25cddab4c0d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:50:43.521682 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.521648 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ad27453-b89c-41db-9111-e84973fc68e0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9ad27453-b89c-41db-9111-e84973fc68e0" (UID: "9ad27453-b89c-41db-9111-e84973fc68e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:50:43.524222 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.523866 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c4dce57-35c7-49a7-886f-25cddab4c0d6-kube-api-access-ffxgs" (OuterVolumeSpecName: "kube-api-access-ffxgs") pod "1c4dce57-35c7-49a7-886f-25cddab4c0d6" (UID: "1c4dce57-35c7-49a7-886f-25cddab4c0d6"). InnerVolumeSpecName "kube-api-access-ffxgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:50:43.524222 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.523923 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad27453-b89c-41db-9111-e84973fc68e0-kube-api-access-gqbx6" (OuterVolumeSpecName: "kube-api-access-gqbx6") pod "9ad27453-b89c-41db-9111-e84973fc68e0" (UID: "9ad27453-b89c-41db-9111-e84973fc68e0"). InnerVolumeSpecName "kube-api-access-gqbx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:50:43.573358 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.573324 2578 generic.go:356] "Generic (PLEG): container finished" podID="9ad27453-b89c-41db-9111-e84973fc68e0" containerID="ee29db4a74d502c14b82d479bcce6c9c43ff9135dfa2c89be36fc626941a0381" exitCode=0 Sep 10 23:50:43.573516 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.573365 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-643c297d-65cb59db7-2l894" Sep 10 23:50:43.573516 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.573409 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-643c297d-65cb59db7-2l894" event={"ID":"9ad27453-b89c-41db-9111-e84973fc68e0","Type":"ContainerDied","Data":"ee29db4a74d502c14b82d479bcce6c9c43ff9135dfa2c89be36fc626941a0381"} Sep 10 23:50:43.573516 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.573457 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-643c297d-65cb59db7-2l894" event={"ID":"9ad27453-b89c-41db-9111-e84973fc68e0","Type":"ContainerDied","Data":"b481c3713b10b4b70960190eefa7b63b82963ddefc6d204094276145cc196022"} Sep 10 23:50:43.573516 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.573483 2578 scope.go:117] "RemoveContainer" containerID="ee29db4a74d502c14b82d479bcce6c9c43ff9135dfa2c89be36fc626941a0381" Sep 10 23:50:43.574800 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.574775 2578 generic.go:356] "Generic (PLEG): container finished" podID="1c4dce57-35c7-49a7-886f-25cddab4c0d6" containerID="6d4be29f3571c1fe52eaffb2846bd42fa3e8742f15fae224f607fee6be817e4d" exitCode=0 Sep 10 23:50:43.574910 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.574817 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh" Sep 10 23:50:43.574910 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.574848 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh" event={"ID":"1c4dce57-35c7-49a7-886f-25cddab4c0d6","Type":"ContainerDied","Data":"6d4be29f3571c1fe52eaffb2846bd42fa3e8742f15fae224f607fee6be817e4d"} Sep 10 23:50:43.574910 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.574903 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh" event={"ID":"1c4dce57-35c7-49a7-886f-25cddab4c0d6","Type":"ContainerDied","Data":"4c9db912b6f547e838fb48eb541557138a32d34bc642ef0e12674010263843d3"} Sep 10 23:50:43.587129 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.587110 2578 scope.go:117] "RemoveContainer" containerID="ee29db4a74d502c14b82d479bcce6c9c43ff9135dfa2c89be36fc626941a0381" Sep 10 23:50:43.587628 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:50:43.587469 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee29db4a74d502c14b82d479bcce6c9c43ff9135dfa2c89be36fc626941a0381\": container with ID starting with ee29db4a74d502c14b82d479bcce6c9c43ff9135dfa2c89be36fc626941a0381 not found: ID does not exist" containerID="ee29db4a74d502c14b82d479bcce6c9c43ff9135dfa2c89be36fc626941a0381" Sep 10 23:50:43.587628 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.587512 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee29db4a74d502c14b82d479bcce6c9c43ff9135dfa2c89be36fc626941a0381"} err="failed to get container status \"ee29db4a74d502c14b82d479bcce6c9c43ff9135dfa2c89be36fc626941a0381\": rpc error: code = NotFound desc = could not find container \"ee29db4a74d502c14b82d479bcce6c9c43ff9135dfa2c89be36fc626941a0381\": container with ID starting with ee29db4a74d502c14b82d479bcce6c9c43ff9135dfa2c89be36fc626941a0381 not found: ID does not exist" Sep 10 23:50:43.587628 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.587537 2578 scope.go:117] "RemoveContainer" containerID="6d4be29f3571c1fe52eaffb2846bd42fa3e8742f15fae224f607fee6be817e4d" Sep 10 23:50:43.597679 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.597657 2578 scope.go:117] "RemoveContainer" containerID="6d4be29f3571c1fe52eaffb2846bd42fa3e8742f15fae224f607fee6be817e4d" Sep 10 23:50:43.598026 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:50:43.598005 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d4be29f3571c1fe52eaffb2846bd42fa3e8742f15fae224f607fee6be817e4d\": container with ID starting with 6d4be29f3571c1fe52eaffb2846bd42fa3e8742f15fae224f607fee6be817e4d not found: ID does not exist" containerID="6d4be29f3571c1fe52eaffb2846bd42fa3e8742f15fae224f607fee6be817e4d" Sep 10 23:50:43.598114 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.598032 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d4be29f3571c1fe52eaffb2846bd42fa3e8742f15fae224f607fee6be817e4d"} err="failed to get container status \"6d4be29f3571c1fe52eaffb2846bd42fa3e8742f15fae224f607fee6be817e4d\": rpc error: code = NotFound desc = could not find container \"6d4be29f3571c1fe52eaffb2846bd42fa3e8742f15fae224f607fee6be817e4d\": container with ID starting with 6d4be29f3571c1fe52eaffb2846bd42fa3e8742f15fae224f607fee6be817e4d not found: ID does not exist" Sep 10 23:50:43.606760 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.606707 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-643c297d-65cb59db7-2l894"] Sep 10 23:50:43.614990 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.614955 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-643c297d-65cb59db7-2l894"] Sep 10 23:50:43.623030 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.623006 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqbx6\" (UniqueName: \"kubernetes.io/projected/9ad27453-b89c-41db-9111-e84973fc68e0-kube-api-access-gqbx6\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:50:43.623125 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.623032 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffxgs\" (UniqueName: \"kubernetes.io/projected/1c4dce57-35c7-49a7-886f-25cddab4c0d6-kube-api-access-ffxgs\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:50:43.623125 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.623046 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad27453-b89c-41db-9111-e84973fc68e0-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:50:43.623125 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.623058 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c4dce57-35c7-49a7-886f-25cddab4c0d6-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:50:43.637773 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.637744 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh"] Sep 10 23:50:43.645175 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:43.645146 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-643c297d-77d6cc756b-zlwjh"] Sep 10 23:50:45.046175 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:45.046141 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c4dce57-35c7-49a7-886f-25cddab4c0d6" path="/var/lib/kubelet/pods/1c4dce57-35c7-49a7-886f-25cddab4c0d6/volumes" Sep 10 23:50:45.046619 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:45.046596 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47e4038b-17bf-41c1-9b9a-7cf91ac24da8" path="/var/lib/kubelet/pods/47e4038b-17bf-41c1-9b9a-7cf91ac24da8/volumes" Sep 10 23:50:45.046930 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:45.046918 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ad27453-b89c-41db-9111-e84973fc68e0" path="/var/lib/kubelet/pods/9ad27453-b89c-41db-9111-e84973fc68e0/volumes" Sep 10 23:50:48.603865 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.603830 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd"] Sep 10 23:50:48.604260 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.604196 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ad27453-b89c-41db-9111-e84973fc68e0" containerName="payload-processing" Sep 10 23:50:48.604260 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.604208 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ad27453-b89c-41db-9111-e84973fc68e0" containerName="payload-processing" Sep 10 23:50:48.604260 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.604222 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c4dce57-35c7-49a7-886f-25cddab4c0d6" containerName="payload-pre-processing" Sep 10 23:50:48.604260 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.604227 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c4dce57-35c7-49a7-886f-25cddab4c0d6" containerName="payload-pre-processing" Sep 10 23:50:48.604260 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.604245 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47e4038b-17bf-41c1-9b9a-7cf91ac24da8" containerName="revoke-keys" Sep 10 23:50:48.604260 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.604251 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="47e4038b-17bf-41c1-9b9a-7cf91ac24da8" containerName="revoke-keys" Sep 10 23:50:48.604438 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.604306 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c4dce57-35c7-49a7-886f-25cddab4c0d6" containerName="payload-pre-processing" Sep 10 23:50:48.604438 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.604317 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ad27453-b89c-41db-9111-e84973fc68e0" containerName="payload-processing" Sep 10 23:50:48.604438 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.604323 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="47e4038b-17bf-41c1-9b9a-7cf91ac24da8" containerName="revoke-keys" Sep 10 23:50:48.608541 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.608523 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd" Sep 10 23:50:48.611389 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.611368 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-58451477-dockercfg-hcch9\"" Sep 10 23:50:48.612271 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.612255 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-58451477\"" Sep 10 23:50:48.628109 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.628086 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd"] Sep 10 23:50:48.718409 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.718366 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-58451477-776b85646-8dbkh"] Sep 10 23:50:48.721913 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.721893 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-58451477-776b85646-8dbkh" Sep 10 23:50:48.739621 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.739599 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-58451477-776b85646-8dbkh"] Sep 10 23:50:48.767593 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.767564 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcfcs\" (UniqueName: \"kubernetes.io/projected/6d4faebe-4015-48b4-9f0e-986cc168fce6-kube-api-access-kcfcs\") pod \"payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd\" (UID: \"6d4faebe-4015-48b4-9f0e-986cc168fce6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd" Sep 10 23:50:48.767722 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.767650 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d4faebe-4015-48b4-9f0e-986cc168fce6-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd\" (UID: \"6d4faebe-4015-48b4-9f0e-986cc168fce6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd" Sep 10 23:50:48.868375 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.868289 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d177ac5-10b5-4cb6-b774-9835363773e5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-58451477-776b85646-8dbkh\" (UID: \"3d177ac5-10b5-4cb6-b774-9835363773e5\") " pod="openshift-ingress/payload-processing-e2e-migrate-58451477-776b85646-8dbkh" Sep 10 23:50:48.868529 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.868403 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d4faebe-4015-48b4-9f0e-986cc168fce6-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd\" (UID: \"6d4faebe-4015-48b4-9f0e-986cc168fce6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd" Sep 10 23:50:48.868529 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.868473 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcfcs\" (UniqueName: \"kubernetes.io/projected/6d4faebe-4015-48b4-9f0e-986cc168fce6-kube-api-access-kcfcs\") pod \"payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd\" (UID: \"6d4faebe-4015-48b4-9f0e-986cc168fce6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd" Sep 10 23:50:48.868529 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.868528 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx6hv\" (UniqueName: \"kubernetes.io/projected/3d177ac5-10b5-4cb6-b774-9835363773e5-kube-api-access-xx6hv\") pod \"payload-processing-e2e-migrate-58451477-776b85646-8dbkh\" (UID: \"3d177ac5-10b5-4cb6-b774-9835363773e5\") " pod="openshift-ingress/payload-processing-e2e-migrate-58451477-776b85646-8dbkh" Sep 10 23:50:48.869183 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.869165 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d4faebe-4015-48b4-9f0e-986cc168fce6-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd\" (UID: \"6d4faebe-4015-48b4-9f0e-986cc168fce6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd" Sep 10 23:50:48.881584 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.881561 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcfcs\" (UniqueName: \"kubernetes.io/projected/6d4faebe-4015-48b4-9f0e-986cc168fce6-kube-api-access-kcfcs\") pod \"payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd\" (UID: \"6d4faebe-4015-48b4-9f0e-986cc168fce6\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd" Sep 10 23:50:48.918579 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.918544 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd" Sep 10 23:50:48.969858 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.969529 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d177ac5-10b5-4cb6-b774-9835363773e5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-58451477-776b85646-8dbkh\" (UID: \"3d177ac5-10b5-4cb6-b774-9835363773e5\") " pod="openshift-ingress/payload-processing-e2e-migrate-58451477-776b85646-8dbkh" Sep 10 23:50:48.970355 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.970319 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx6hv\" (UniqueName: \"kubernetes.io/projected/3d177ac5-10b5-4cb6-b774-9835363773e5-kube-api-access-xx6hv\") pod \"payload-processing-e2e-migrate-58451477-776b85646-8dbkh\" (UID: \"3d177ac5-10b5-4cb6-b774-9835363773e5\") " pod="openshift-ingress/payload-processing-e2e-migrate-58451477-776b85646-8dbkh" Sep 10 23:50:48.970481 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.970456 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d177ac5-10b5-4cb6-b774-9835363773e5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-58451477-776b85646-8dbkh\" (UID: \"3d177ac5-10b5-4cb6-b774-9835363773e5\") " pod="openshift-ingress/payload-processing-e2e-migrate-58451477-776b85646-8dbkh" Sep 10 23:50:48.981051 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:48.980991 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx6hv\" (UniqueName: \"kubernetes.io/projected/3d177ac5-10b5-4cb6-b774-9835363773e5-kube-api-access-xx6hv\") pod \"payload-processing-e2e-migrate-58451477-776b85646-8dbkh\" (UID: \"3d177ac5-10b5-4cb6-b774-9835363773e5\") " pod="openshift-ingress/payload-processing-e2e-migrate-58451477-776b85646-8dbkh" Sep 10 23:50:49.032378 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:49.032350 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-58451477-776b85646-8dbkh" Sep 10 23:50:49.191414 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:49.191389 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd"] Sep 10 23:50:49.193203 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:50:49.193174 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d4faebe_4015_48b4_9f0e_986cc168fce6.slice/crio-49268b49012b6c1eefc009e0f34b45de0c1bff9fe76af7d87224fd456a173840 WatchSource:0}: Error finding container 49268b49012b6c1eefc009e0f34b45de0c1bff9fe76af7d87224fd456a173840: Status 404 returned error can't find the container with id 49268b49012b6c1eefc009e0f34b45de0c1bff9fe76af7d87224fd456a173840 Sep 10 23:50:49.303645 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:49.303572 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-58451477-776b85646-8dbkh"] Sep 10 23:50:49.305814 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:50:49.305789 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d177ac5_10b5_4cb6_b774_9835363773e5.slice/crio-4920584444a3ab1160ac11e4f5aedf9f10ecb83ee48418174606790fa3ca6a5d WatchSource:0}: Error finding container 4920584444a3ab1160ac11e4f5aedf9f10ecb83ee48418174606790fa3ca6a5d: Status 404 returned error can't find the container with id 4920584444a3ab1160ac11e4f5aedf9f10ecb83ee48418174606790fa3ca6a5d Sep 10 23:50:49.609387 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:49.609274 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd" event={"ID":"6d4faebe-4015-48b4-9f0e-986cc168fce6","Type":"ContainerStarted","Data":"3dc9dc06527789dd0d07dbd08f50b3a3467f21abfb12ba774981be7dd9e07acf"} Sep 10 23:50:49.609387 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:49.609352 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd" Sep 10 23:50:49.609387 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:49.609379 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd" event={"ID":"6d4faebe-4015-48b4-9f0e-986cc168fce6","Type":"ContainerStarted","Data":"49268b49012b6c1eefc009e0f34b45de0c1bff9fe76af7d87224fd456a173840"} Sep 10 23:50:49.610863 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:49.610838 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-58451477-776b85646-8dbkh" event={"ID":"3d177ac5-10b5-4cb6-b774-9835363773e5","Type":"ContainerStarted","Data":"34526312c5f27594e73044c83f7a0a94cf31d7e436e56c89cd812cc9fe61984b"} Sep 10 23:50:49.610969 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:49.610886 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-58451477-776b85646-8dbkh" event={"ID":"3d177ac5-10b5-4cb6-b774-9835363773e5","Type":"ContainerStarted","Data":"4920584444a3ab1160ac11e4f5aedf9f10ecb83ee48418174606790fa3ca6a5d"} Sep 10 23:50:49.611009 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:49.610961 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-58451477-776b85646-8dbkh" Sep 10 23:50:49.626489 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:49.626445 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd" podStartSLOduration=1.626431137 podStartE2EDuration="1.626431137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:49.625614794 +0000 UTC m=+865.093089132" watchObservedRunningTime="2026-09-10 23:50:49.626431137 +0000 UTC m=+865.093905461" Sep 10 23:50:49.643885 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:50:49.643840 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-58451477-776b85646-8dbkh" podStartSLOduration=1.643826081 podStartE2EDuration="1.643826081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:49.642494731 +0000 UTC m=+865.109969056" watchObservedRunningTime="2026-09-10 23:50:49.643826081 +0000 UTC m=+865.111300423" Sep 10 23:51:00.617703 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:00.617650 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd" Sep 10 23:51:00.618226 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:00.617974 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-58451477-776b85646-8dbkh" Sep 10 23:51:25.004074 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:25.003991 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zkr6f_aacd921b-5c15-4f7f-8685-cfceebfe5407/console-operator/2.log" Sep 10 23:51:25.004800 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:25.004778 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zkr6f_aacd921b-5c15-4f7f-8685-cfceebfe5407/console-operator/2.log" Sep 10 23:51:39.721180 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:39.721140 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-58451477-776b85646-8dbkh"] Sep 10 23:51:39.721810 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:39.721392 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-58451477-776b85646-8dbkh" podUID="3d177ac5-10b5-4cb6-b774-9835363773e5" containerName="payload-processing" containerID="cri-o://34526312c5f27594e73044c83f7a0a94cf31d7e436e56c89cd812cc9fe61984b" gracePeriod=30 Sep 10 23:51:39.727996 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:39.727967 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd"] Sep 10 23:51:39.728212 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:39.728182 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd" podUID="6d4faebe-4015-48b4-9f0e-986cc168fce6" containerName="payload-pre-processing" containerID="cri-o://3dc9dc06527789dd0d07dbd08f50b3a3467f21abfb12ba774981be7dd9e07acf" gracePeriod=30 Sep 10 23:51:39.849757 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:39.849722 2578 generic.go:356] "Generic (PLEG): container finished" podID="6d4faebe-4015-48b4-9f0e-986cc168fce6" containerID="3dc9dc06527789dd0d07dbd08f50b3a3467f21abfb12ba774981be7dd9e07acf" exitCode=0 Sep 10 23:51:39.849895 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:39.849795 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd" event={"ID":"6d4faebe-4015-48b4-9f0e-986cc168fce6","Type":"ContainerDied","Data":"3dc9dc06527789dd0d07dbd08f50b3a3467f21abfb12ba774981be7dd9e07acf"} Sep 10 23:51:39.851606 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:39.851581 2578 generic.go:356] "Generic (PLEG): container finished" podID="3d177ac5-10b5-4cb6-b774-9835363773e5" containerID="34526312c5f27594e73044c83f7a0a94cf31d7e436e56c89cd812cc9fe61984b" exitCode=0 Sep 10 23:51:39.851706 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:39.851659 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-58451477-776b85646-8dbkh" event={"ID":"3d177ac5-10b5-4cb6-b774-9835363773e5","Type":"ContainerDied","Data":"34526312c5f27594e73044c83f7a0a94cf31d7e436e56c89cd812cc9fe61984b"} Sep 10 23:51:40.009892 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.009861 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-58451477-776b85646-8dbkh" Sep 10 23:51:40.013341 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.013317 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd" Sep 10 23:51:40.133068 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.133035 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d177ac5-10b5-4cb6-b774-9835363773e5-plugins-config-volume\") pod \"3d177ac5-10b5-4cb6-b774-9835363773e5\" (UID: \"3d177ac5-10b5-4cb6-b774-9835363773e5\") " Sep 10 23:51:40.133251 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.133123 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcfcs\" (UniqueName: \"kubernetes.io/projected/6d4faebe-4015-48b4-9f0e-986cc168fce6-kube-api-access-kcfcs\") pod \"6d4faebe-4015-48b4-9f0e-986cc168fce6\" (UID: \"6d4faebe-4015-48b4-9f0e-986cc168fce6\") " Sep 10 23:51:40.133251 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.133157 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d4faebe-4015-48b4-9f0e-986cc168fce6-plugins-config-volume\") pod \"6d4faebe-4015-48b4-9f0e-986cc168fce6\" (UID: \"6d4faebe-4015-48b4-9f0e-986cc168fce6\") " Sep 10 23:51:40.133251 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.133179 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xx6hv\" (UniqueName: \"kubernetes.io/projected/3d177ac5-10b5-4cb6-b774-9835363773e5-kube-api-access-xx6hv\") pod \"3d177ac5-10b5-4cb6-b774-9835363773e5\" (UID: \"3d177ac5-10b5-4cb6-b774-9835363773e5\") " Sep 10 23:51:40.133493 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.133462 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d177ac5-10b5-4cb6-b774-9835363773e5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3d177ac5-10b5-4cb6-b774-9835363773e5" (UID: "3d177ac5-10b5-4cb6-b774-9835363773e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:51:40.133561 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.133518 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d4faebe-4015-48b4-9f0e-986cc168fce6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6d4faebe-4015-48b4-9f0e-986cc168fce6" (UID: "6d4faebe-4015-48b4-9f0e-986cc168fce6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:51:40.135458 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.135425 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d177ac5-10b5-4cb6-b774-9835363773e5-kube-api-access-xx6hv" (OuterVolumeSpecName: "kube-api-access-xx6hv") pod "3d177ac5-10b5-4cb6-b774-9835363773e5" (UID: "3d177ac5-10b5-4cb6-b774-9835363773e5"). InnerVolumeSpecName "kube-api-access-xx6hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:51:40.135552 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.135500 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d4faebe-4015-48b4-9f0e-986cc168fce6-kube-api-access-kcfcs" (OuterVolumeSpecName: "kube-api-access-kcfcs") pod "6d4faebe-4015-48b4-9f0e-986cc168fce6" (UID: "6d4faebe-4015-48b4-9f0e-986cc168fce6"). InnerVolumeSpecName "kube-api-access-kcfcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:51:40.234045 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.233957 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d177ac5-10b5-4cb6-b774-9835363773e5-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:51:40.234045 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.233992 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcfcs\" (UniqueName: \"kubernetes.io/projected/6d4faebe-4015-48b4-9f0e-986cc168fce6-kube-api-access-kcfcs\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:51:40.234045 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.234002 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d4faebe-4015-48b4-9f0e-986cc168fce6-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:51:40.234045 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.234010 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx6hv\" (UniqueName: \"kubernetes.io/projected/3d177ac5-10b5-4cb6-b774-9835363773e5-kube-api-access-xx6hv\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:51:40.761658 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.761621 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk"] Sep 10 23:51:40.764102 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.764071 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d177ac5-10b5-4cb6-b774-9835363773e5" containerName="payload-processing" Sep 10 23:51:40.764235 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.764110 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d177ac5-10b5-4cb6-b774-9835363773e5" containerName="payload-processing" Sep 10 23:51:40.764235 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.764126 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d4faebe-4015-48b4-9f0e-986cc168fce6" containerName="payload-pre-processing" Sep 10 23:51:40.764235 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.764136 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d4faebe-4015-48b4-9f0e-986cc168fce6" containerName="payload-pre-processing" Sep 10 23:51:40.764337 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.764275 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d177ac5-10b5-4cb6-b774-9835363773e5" containerName="payload-processing" Sep 10 23:51:40.764337 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.764288 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d4faebe-4015-48b4-9f0e-986cc168fce6" containerName="payload-pre-processing" Sep 10 23:51:40.767415 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.767398 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:40.769843 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.769823 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f9a25dcc-openshift-default-dockercfg-8f9cf\"" Sep 10 23:51:40.775722 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.775694 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk"] Sep 10 23:51:40.868135 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.868105 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd" Sep 10 23:51:40.868325 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.868105 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd" event={"ID":"6d4faebe-4015-48b4-9f0e-986cc168fce6","Type":"ContainerDied","Data":"49268b49012b6c1eefc009e0f34b45de0c1bff9fe76af7d87224fd456a173840"} Sep 10 23:51:40.868325 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.868259 2578 scope.go:117] "RemoveContainer" containerID="3dc9dc06527789dd0d07dbd08f50b3a3467f21abfb12ba774981be7dd9e07acf" Sep 10 23:51:40.870218 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.870195 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-58451477-776b85646-8dbkh" event={"ID":"3d177ac5-10b5-4cb6-b774-9835363773e5","Type":"ContainerDied","Data":"4920584444a3ab1160ac11e4f5aedf9f10ecb83ee48418174606790fa3ca6a5d"} Sep 10 23:51:40.870302 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.870288 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-58451477-776b85646-8dbkh" Sep 10 23:51:40.882308 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.882211 2578 scope.go:117] "RemoveContainer" containerID="34526312c5f27594e73044c83f7a0a94cf31d7e436e56c89cd812cc9fe61984b" Sep 10 23:51:40.894230 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.894207 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd"] Sep 10 23:51:40.898310 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.898288 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-58451477-7d465dc89d-jpvbd"] Sep 10 23:51:40.908711 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.908689 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-58451477-776b85646-8dbkh"] Sep 10 23:51:40.912814 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.912794 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-58451477-776b85646-8dbkh"] Sep 10 23:51:40.940708 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.940685 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0558f6dd-9393-4fbc-938e-78699eecb86f-istiod-ca-cert\") pod \"e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:40.940806 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.940727 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-istio-data\") pod \"e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:40.940806 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.940771 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0558f6dd-9393-4fbc-938e-78699eecb86f-istio-podinfo\") pod \"e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:40.940889 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.940818 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-workload-socket\") pod \"e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:40.940889 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.940833 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-workload-certs\") pod \"e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:40.940985 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.940879 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0558f6dd-9393-4fbc-938e-78699eecb86f-istio-token\") pod \"e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:40.940985 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.940967 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x827f\" (UniqueName: \"kubernetes.io/projected/0558f6dd-9393-4fbc-938e-78699eecb86f-kube-api-access-x827f\") pod \"e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:40.941075 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.941060 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-credential-socket\") pod \"e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:40.941121 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:40.941092 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-istio-envoy\") pod \"e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:41.041691 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.041588 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-workload-socket\") pod \"e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:41.041691 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.041649 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-workload-certs\") pod \"e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:41.041691 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.041674 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0558f6dd-9393-4fbc-938e-78699eecb86f-istio-token\") pod \"e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:41.041991 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.041724 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x827f\" (UniqueName: \"kubernetes.io/projected/0558f6dd-9393-4fbc-938e-78699eecb86f-kube-api-access-x827f\") pod \"e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:41.041991 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.041797 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-credential-socket\") pod \"e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:41.041991 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.041826 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-istio-envoy\") pod \"e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:41.041991 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.041871 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0558f6dd-9393-4fbc-938e-78699eecb86f-istiod-ca-cert\") pod \"e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:41.041991 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.041916 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-istio-data\") pod \"e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:41.041991 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.041969 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0558f6dd-9393-4fbc-938e-78699eecb86f-istio-podinfo\") pod \"e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:41.042280 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.042063 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-workload-socket\") pod \"e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:41.042342 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.042321 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-workload-certs\") pod \"e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:41.042547 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.042527 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-credential-socket\") pod \"e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:41.042862 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.042838 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-istio-data\") pod \"e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:41.043101 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.043080 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0558f6dd-9393-4fbc-938e-78699eecb86f-istiod-ca-cert\") pod \"e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:41.044780 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.044748 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-istio-envoy\") pod \"e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:41.044878 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.044847 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0558f6dd-9393-4fbc-938e-78699eecb86f-istio-podinfo\") pod \"e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:41.046290 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.046268 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d177ac5-10b5-4cb6-b774-9835363773e5" path="/var/lib/kubelet/pods/3d177ac5-10b5-4cb6-b774-9835363773e5/volumes" Sep 10 23:51:41.046621 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.046607 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d4faebe-4015-48b4-9f0e-986cc168fce6" path="/var/lib/kubelet/pods/6d4faebe-4015-48b4-9f0e-986cc168fce6/volumes" Sep 10 23:51:41.052389 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.052367 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x827f\" (UniqueName: \"kubernetes.io/projected/0558f6dd-9393-4fbc-938e-78699eecb86f-kube-api-access-x827f\") pod \"e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:41.052496 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.052416 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0558f6dd-9393-4fbc-938e-78699eecb86f-istio-token\") pod \"e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:41.082451 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.082420 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:41.357657 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.357634 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk"] Sep 10 23:51:41.359332 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:51:41.359301 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0558f6dd_9393_4fbc_938e_78699eecb86f.slice/crio-236e0d5cd2704486ee1e52388d4861c63d4395054e2535f6c2e13d3b648845ee WatchSource:0}: Error finding container 236e0d5cd2704486ee1e52388d4861c63d4395054e2535f6c2e13d3b648845ee: Status 404 returned error can't find the container with id 236e0d5cd2704486ee1e52388d4861c63d4395054e2535f6c2e13d3b648845ee Sep 10 23:51:41.361476 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.361448 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:51:41.361582 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.361522 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:51:41.361582 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.361571 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:51:41.877160 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.877122 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" event={"ID":"0558f6dd-9393-4fbc-938e-78699eecb86f","Type":"ContainerStarted","Data":"be9a6f6d54cc9948f57281448b1e57259386d5e77e57380917eee86065a4b69d"} Sep 10 23:51:41.877633 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.877183 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" event={"ID":"0558f6dd-9393-4fbc-938e-78699eecb86f","Type":"ContainerStarted","Data":"236e0d5cd2704486ee1e52388d4861c63d4395054e2535f6c2e13d3b648845ee"} Sep 10 23:51:41.900379 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:41.900323 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" podStartSLOduration=1.900306741 podStartE2EDuration="1.900306741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:51:41.898426306 +0000 UTC m=+917.365900633" watchObservedRunningTime="2026-09-10 23:51:41.900306741 +0000 UTC m=+917.367781066" Sep 10 23:51:42.083125 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:42.083087 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:42.088333 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:42.088310 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:42.883361 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:42.883332 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:42.884335 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:42.884315 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:51:45.354272 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:45.354236 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2"] Sep 10 23:51:45.357664 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:45.357643 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2" Sep 10 23:51:45.360182 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:45.360165 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f9a25dcc\"" Sep 10 23:51:45.368819 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:45.368791 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2"] Sep 10 23:51:45.433210 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:45.433175 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt"] Sep 10 23:51:45.436462 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:45.436445 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt" Sep 10 23:51:45.438783 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:45.438764 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f9a25dcc-dockercfg-8sj7b\"" Sep 10 23:51:45.448019 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:45.447997 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt"] Sep 10 23:51:45.489964 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:45.489914 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8rtk\" (UniqueName: \"kubernetes.io/projected/8dc0bab2-af48-4152-9df9-189943fb1673-kube-api-access-z8rtk\") pod \"payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2\" (UID: \"8dc0bab2-af48-4152-9df9-189943fb1673\") " pod="openshift-ingress/payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2" Sep 10 23:51:45.490138 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:45.490081 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dc0bab2-af48-4152-9df9-189943fb1673-plugins-config-volume\") pod \"payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2\" (UID: \"8dc0bab2-af48-4152-9df9-189943fb1673\") " pod="openshift-ingress/payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2" Sep 10 23:51:45.591421 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:45.591382 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dc0bab2-af48-4152-9df9-189943fb1673-plugins-config-volume\") pod \"payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2\" (UID: \"8dc0bab2-af48-4152-9df9-189943fb1673\") " pod="openshift-ingress/payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2" Sep 10 23:51:45.591593 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:45.591482 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/411c7bb1-d006-46c0-882a-aeb3bb9a6904-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt\" (UID: \"411c7bb1-d006-46c0-882a-aeb3bb9a6904\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt" Sep 10 23:51:45.591593 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:45.591522 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8rtk\" (UniqueName: \"kubernetes.io/projected/8dc0bab2-af48-4152-9df9-189943fb1673-kube-api-access-z8rtk\") pod \"payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2\" (UID: \"8dc0bab2-af48-4152-9df9-189943fb1673\") " pod="openshift-ingress/payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2" Sep 10 23:51:45.591593 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:45.591549 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7vbx\" (UniqueName: \"kubernetes.io/projected/411c7bb1-d006-46c0-882a-aeb3bb9a6904-kube-api-access-m7vbx\") pod \"payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt\" (UID: \"411c7bb1-d006-46c0-882a-aeb3bb9a6904\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt" Sep 10 23:51:45.592124 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:45.592104 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dc0bab2-af48-4152-9df9-189943fb1673-plugins-config-volume\") pod \"payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2\" (UID: \"8dc0bab2-af48-4152-9df9-189943fb1673\") " pod="openshift-ingress/payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2" Sep 10 23:51:45.602096 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:45.602073 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8rtk\" (UniqueName: \"kubernetes.io/projected/8dc0bab2-af48-4152-9df9-189943fb1673-kube-api-access-z8rtk\") pod \"payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2\" (UID: \"8dc0bab2-af48-4152-9df9-189943fb1673\") " pod="openshift-ingress/payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2" Sep 10 23:51:45.670093 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:45.670056 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2" Sep 10 23:51:45.692536 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:45.692505 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/411c7bb1-d006-46c0-882a-aeb3bb9a6904-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt\" (UID: \"411c7bb1-d006-46c0-882a-aeb3bb9a6904\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt" Sep 10 23:51:45.692688 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:45.692556 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7vbx\" (UniqueName: \"kubernetes.io/projected/411c7bb1-d006-46c0-882a-aeb3bb9a6904-kube-api-access-m7vbx\") pod \"payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt\" (UID: \"411c7bb1-d006-46c0-882a-aeb3bb9a6904\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt" Sep 10 23:51:45.693187 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:45.693166 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/411c7bb1-d006-46c0-882a-aeb3bb9a6904-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt\" (UID: \"411c7bb1-d006-46c0-882a-aeb3bb9a6904\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt" Sep 10 23:51:45.700804 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:45.700771 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7vbx\" (UniqueName: \"kubernetes.io/projected/411c7bb1-d006-46c0-882a-aeb3bb9a6904-kube-api-access-m7vbx\") pod \"payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt\" (UID: \"411c7bb1-d006-46c0-882a-aeb3bb9a6904\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt" Sep 10 23:51:45.746319 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:45.746286 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt" Sep 10 23:51:45.941311 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:45.941284 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2"] Sep 10 23:51:45.943013 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:51:45.942989 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dc0bab2_af48_4152_9df9_189943fb1673.slice/crio-04b82fa666bffb015281e5662bc0fe45b5a80d5071b442324c71cdcd3c8510db WatchSource:0}: Error finding container 04b82fa666bffb015281e5662bc0fe45b5a80d5071b442324c71cdcd3c8510db: Status 404 returned error can't find the container with id 04b82fa666bffb015281e5662bc0fe45b5a80d5071b442324c71cdcd3c8510db Sep 10 23:51:46.027579 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:46.027544 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt"] Sep 10 23:51:46.028792 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:51:46.028755 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod411c7bb1_d006_46c0_882a_aeb3bb9a6904.slice/crio-44871f5205723d455f69124f99065e08b1c83d09c32bfcbf2c1bece1f85af167 WatchSource:0}: Error finding container 44871f5205723d455f69124f99065e08b1c83d09c32bfcbf2c1bece1f85af167: Status 404 returned error can't find the container with id 44871f5205723d455f69124f99065e08b1c83d09c32bfcbf2c1bece1f85af167 Sep 10 23:51:46.903004 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:46.902967 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2" event={"ID":"8dc0bab2-af48-4152-9df9-189943fb1673","Type":"ContainerStarted","Data":"ca8c61afd281fffc4da78bb154cdcbdc7f9e4c56d7c4860ad97bca25e45f91c8"} Sep 10 23:51:46.903478 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:46.903030 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2" event={"ID":"8dc0bab2-af48-4152-9df9-189943fb1673","Type":"ContainerStarted","Data":"04b82fa666bffb015281e5662bc0fe45b5a80d5071b442324c71cdcd3c8510db"} Sep 10 23:51:46.903478 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:46.903152 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2" Sep 10 23:51:46.904686 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:46.904663 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt" event={"ID":"411c7bb1-d006-46c0-882a-aeb3bb9a6904","Type":"ContainerStarted","Data":"170659ce4a9ef6f750a338344fdbb64a93931f5d36cd1d2bbe506c5cf9dbdfb7"} Sep 10 23:51:46.904824 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:46.904709 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt" event={"ID":"411c7bb1-d006-46c0-882a-aeb3bb9a6904","Type":"ContainerStarted","Data":"44871f5205723d455f69124f99065e08b1c83d09c32bfcbf2c1bece1f85af167"} Sep 10 23:51:46.904824 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:46.904740 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt" Sep 10 23:51:46.921949 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:46.921864 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2" podStartSLOduration=1.921845916 podStartE2EDuration="1.921845916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:51:46.921084294 +0000 UTC m=+922.388558613" watchObservedRunningTime="2026-09-10 23:51:46.921845916 +0000 UTC m=+922.389320242" Sep 10 23:51:46.938551 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:46.938495 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt" podStartSLOduration=1.938476332 podStartE2EDuration="1.938476332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:51:46.936441125 +0000 UTC m=+922.403915481" watchObservedRunningTime="2026-09-10 23:51:46.938476332 +0000 UTC m=+922.405950657" Sep 10 23:51:57.911621 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:57.911588 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt" Sep 10 23:51:57.912139 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:51:57.911671 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2" Sep 10 23:52:06.695045 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:06.695007 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg"] Sep 10 23:52:06.695482 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:06.695235 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg" podUID="5aab5922-dba3-4f74-a65d-63b61dc1e138" containerName="payload-pre-processing" containerID="cri-o://29ab92a12fad5767861f4b78705a9a05af1ed40e0ebecfb298b03ec45a8e6cc7" gracePeriod=30 Sep 10 23:52:06.988660 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:06.988592 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg" Sep 10 23:52:07.007025 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:07.006841 2578 generic.go:356] "Generic (PLEG): container finished" podID="5aab5922-dba3-4f74-a65d-63b61dc1e138" containerID="29ab92a12fad5767861f4b78705a9a05af1ed40e0ebecfb298b03ec45a8e6cc7" exitCode=0 Sep 10 23:52:07.007355 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:07.007328 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg" event={"ID":"5aab5922-dba3-4f74-a65d-63b61dc1e138","Type":"ContainerDied","Data":"29ab92a12fad5767861f4b78705a9a05af1ed40e0ebecfb298b03ec45a8e6cc7"} Sep 10 23:52:07.007442 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:07.007391 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg" event={"ID":"5aab5922-dba3-4f74-a65d-63b61dc1e138","Type":"ContainerDied","Data":"728808ab4d87633d29ad5206a62308d0b808f052b0b3828262e8c493820c0872"} Sep 10 23:52:07.007442 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:07.007422 2578 scope.go:117] "RemoveContainer" containerID="29ab92a12fad5767861f4b78705a9a05af1ed40e0ebecfb298b03ec45a8e6cc7" Sep 10 23:52:07.007567 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:07.007546 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg" Sep 10 23:52:07.016899 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:07.016880 2578 scope.go:117] "RemoveContainer" containerID="29ab92a12fad5767861f4b78705a9a05af1ed40e0ebecfb298b03ec45a8e6cc7" Sep 10 23:52:07.017215 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:52:07.017190 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29ab92a12fad5767861f4b78705a9a05af1ed40e0ebecfb298b03ec45a8e6cc7\": container with ID starting with 29ab92a12fad5767861f4b78705a9a05af1ed40e0ebecfb298b03ec45a8e6cc7 not found: ID does not exist" containerID="29ab92a12fad5767861f4b78705a9a05af1ed40e0ebecfb298b03ec45a8e6cc7" Sep 10 23:52:07.017271 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:07.017220 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ab92a12fad5767861f4b78705a9a05af1ed40e0ebecfb298b03ec45a8e6cc7"} err="failed to get container status \"29ab92a12fad5767861f4b78705a9a05af1ed40e0ebecfb298b03ec45a8e6cc7\": rpc error: code = NotFound desc = could not find container \"29ab92a12fad5767861f4b78705a9a05af1ed40e0ebecfb298b03ec45a8e6cc7\": container with ID starting with 29ab92a12fad5767861f4b78705a9a05af1ed40e0ebecfb298b03ec45a8e6cc7 not found: ID does not exist" Sep 10 23:52:07.091510 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:07.091479 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aab5922-dba3-4f74-a65d-63b61dc1e138-plugins-config-volume\") pod \"5aab5922-dba3-4f74-a65d-63b61dc1e138\" (UID: \"5aab5922-dba3-4f74-a65d-63b61dc1e138\") " Sep 10 23:52:07.091675 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:07.091518 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqkpx\" (UniqueName: \"kubernetes.io/projected/5aab5922-dba3-4f74-a65d-63b61dc1e138-kube-api-access-hqkpx\") pod \"5aab5922-dba3-4f74-a65d-63b61dc1e138\" (UID: \"5aab5922-dba3-4f74-a65d-63b61dc1e138\") " Sep 10 23:52:07.091833 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:07.091807 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aab5922-dba3-4f74-a65d-63b61dc1e138-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5aab5922-dba3-4f74-a65d-63b61dc1e138" (UID: "5aab5922-dba3-4f74-a65d-63b61dc1e138"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:52:07.093638 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:07.093610 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aab5922-dba3-4f74-a65d-63b61dc1e138-kube-api-access-hqkpx" (OuterVolumeSpecName: "kube-api-access-hqkpx") pod "5aab5922-dba3-4f74-a65d-63b61dc1e138" (UID: "5aab5922-dba3-4f74-a65d-63b61dc1e138"). InnerVolumeSpecName "kube-api-access-hqkpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:52:07.192869 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:07.192833 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aab5922-dba3-4f74-a65d-63b61dc1e138-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:52:07.192869 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:07.192865 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqkpx\" (UniqueName: \"kubernetes.io/projected/5aab5922-dba3-4f74-a65d-63b61dc1e138-kube-api-access-hqkpx\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:52:07.330180 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:07.330149 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg"] Sep 10 23:52:07.334042 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:07.334015 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-66836f-6cdb6c8b58-hfxqg"] Sep 10 23:52:09.044421 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:09.044383 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5aab5922-dba3-4f74-a65d-63b61dc1e138" path="/var/lib/kubelet/pods/5aab5922-dba3-4f74-a65d-63b61dc1e138/volumes" Sep 10 23:52:11.035091 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:11.035052 2578 generic.go:356] "Generic (PLEG): container finished" podID="0558f6dd-9393-4fbc-938e-78699eecb86f" containerID="be9a6f6d54cc9948f57281448b1e57259386d5e77e57380917eee86065a4b69d" exitCode=0 Sep 10 23:52:11.035556 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:11.035117 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" event={"ID":"0558f6dd-9393-4fbc-938e-78699eecb86f","Type":"ContainerDied","Data":"be9a6f6d54cc9948f57281448b1e57259386d5e77e57380917eee86065a4b69d"} Sep 10 23:52:11.035556 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:11.035497 2578 scope.go:117] "RemoveContainer" containerID="be9a6f6d54cc9948f57281448b1e57259386d5e77e57380917eee86065a4b69d" Sep 10 23:52:11.036510 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:11.036482 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:52:11.036632 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:11.036561 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:52:11.036632 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:11.036609 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:52:12.045843 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:12.045803 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" event={"ID":"0558f6dd-9393-4fbc-938e-78699eecb86f","Type":"ContainerStarted","Data":"455107b955ac7753851dd615efb2fc64bb4907005753e3a68ba4ae153c3e6a06"} Sep 10 23:52:12.083387 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:12.083353 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:52:12.084547 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:12.084521 2578 patch_prober.go:28] interesting pod/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Sep 10 23:52:12.084660 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:12.084571 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" podUID="0558f6dd-9393-4fbc-938e-78699eecb86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Sep 10 23:52:12.883502 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:12.883467 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:52:13.082879 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:13.082844 2578 patch_prober.go:28] interesting pod/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Sep 10 23:52:13.083258 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:13.082895 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" podUID="0558f6dd-9393-4fbc-938e-78699eecb86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Sep 10 23:52:14.083454 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:14.083422 2578 patch_prober.go:28] interesting pod/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Sep 10 23:52:14.083805 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:14.083496 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" podUID="0558f6dd-9393-4fbc-938e-78699eecb86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Sep 10 23:52:15.019228 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:15.019190 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2"] Sep 10 23:52:15.019463 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:15.019434 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2" podUID="8dc0bab2-af48-4152-9df9-189943fb1673" containerName="payload-processing" containerID="cri-o://ca8c61afd281fffc4da78bb154cdcbdc7f9e4c56d7c4860ad97bca25e45f91c8" gracePeriod=30 Sep 10 23:52:15.024714 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:15.024688 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt"] Sep 10 23:52:15.025065 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:15.025017 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt" podUID="411c7bb1-d006-46c0-882a-aeb3bb9a6904" containerName="payload-pre-processing" containerID="cri-o://170659ce4a9ef6f750a338344fdbb64a93931f5d36cd1d2bbe506c5cf9dbdfb7" gracePeriod=30 Sep 10 23:52:15.083127 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:15.083086 2578 patch_prober.go:28] interesting pod/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Sep 10 23:52:15.083286 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:15.083156 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" podUID="0558f6dd-9393-4fbc-938e-78699eecb86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Sep 10 23:52:15.314119 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:15.314091 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2" Sep 10 23:52:15.372899 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:15.372743 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt" Sep 10 23:52:15.382794 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:15.382681 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dc0bab2-af48-4152-9df9-189943fb1673-plugins-config-volume\") pod \"8dc0bab2-af48-4152-9df9-189943fb1673\" (UID: \"8dc0bab2-af48-4152-9df9-189943fb1673\") " Sep 10 23:52:15.382794 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:15.382764 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8rtk\" (UniqueName: \"kubernetes.io/projected/8dc0bab2-af48-4152-9df9-189943fb1673-kube-api-access-z8rtk\") pod \"8dc0bab2-af48-4152-9df9-189943fb1673\" (UID: \"8dc0bab2-af48-4152-9df9-189943fb1673\") " Sep 10 23:52:15.383166 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:15.383136 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dc0bab2-af48-4152-9df9-189943fb1673-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8dc0bab2-af48-4152-9df9-189943fb1673" (UID: "8dc0bab2-af48-4152-9df9-189943fb1673"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:52:15.385678 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:15.385645 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dc0bab2-af48-4152-9df9-189943fb1673-kube-api-access-z8rtk" (OuterVolumeSpecName: "kube-api-access-z8rtk") pod "8dc0bab2-af48-4152-9df9-189943fb1673" (UID: "8dc0bab2-af48-4152-9df9-189943fb1673"). InnerVolumeSpecName "kube-api-access-z8rtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:52:15.483840 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:15.483797 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7vbx\" (UniqueName: \"kubernetes.io/projected/411c7bb1-d006-46c0-882a-aeb3bb9a6904-kube-api-access-m7vbx\") pod \"411c7bb1-d006-46c0-882a-aeb3bb9a6904\" (UID: \"411c7bb1-d006-46c0-882a-aeb3bb9a6904\") " Sep 10 23:52:15.484031 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:15.483981 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/411c7bb1-d006-46c0-882a-aeb3bb9a6904-plugins-config-volume\") pod \"411c7bb1-d006-46c0-882a-aeb3bb9a6904\" (UID: \"411c7bb1-d006-46c0-882a-aeb3bb9a6904\") " Sep 10 23:52:15.484226 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:15.484198 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dc0bab2-af48-4152-9df9-189943fb1673-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:52:15.484307 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:15.484225 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8rtk\" (UniqueName: \"kubernetes.io/projected/8dc0bab2-af48-4152-9df9-189943fb1673-kube-api-access-z8rtk\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:52:15.484350 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:15.484331 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/411c7bb1-d006-46c0-882a-aeb3bb9a6904-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "411c7bb1-d006-46c0-882a-aeb3bb9a6904" (UID: "411c7bb1-d006-46c0-882a-aeb3bb9a6904"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:52:15.485869 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:15.485853 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/411c7bb1-d006-46c0-882a-aeb3bb9a6904-kube-api-access-m7vbx" (OuterVolumeSpecName: "kube-api-access-m7vbx") pod "411c7bb1-d006-46c0-882a-aeb3bb9a6904" (UID: "411c7bb1-d006-46c0-882a-aeb3bb9a6904"). InnerVolumeSpecName "kube-api-access-m7vbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:52:15.585206 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:15.585121 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/411c7bb1-d006-46c0-882a-aeb3bb9a6904-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:52:15.585206 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:15.585151 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7vbx\" (UniqueName: \"kubernetes.io/projected/411c7bb1-d006-46c0-882a-aeb3bb9a6904-kube-api-access-m7vbx\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:52:16.083535 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:16.083496 2578 patch_prober.go:28] interesting pod/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Sep 10 23:52:16.083706 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:16.083555 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" podUID="0558f6dd-9393-4fbc-938e-78699eecb86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Sep 10 23:52:16.091997 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:16.091966 2578 generic.go:356] "Generic (PLEG): container finished" podID="8dc0bab2-af48-4152-9df9-189943fb1673" containerID="ca8c61afd281fffc4da78bb154cdcbdc7f9e4c56d7c4860ad97bca25e45f91c8" exitCode=0 Sep 10 23:52:16.092146 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:16.092004 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2" Sep 10 23:52:16.092146 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:16.092028 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2" event={"ID":"8dc0bab2-af48-4152-9df9-189943fb1673","Type":"ContainerDied","Data":"ca8c61afd281fffc4da78bb154cdcbdc7f9e4c56d7c4860ad97bca25e45f91c8"} Sep 10 23:52:16.092146 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:16.092078 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2" event={"ID":"8dc0bab2-af48-4152-9df9-189943fb1673","Type":"ContainerDied","Data":"04b82fa666bffb015281e5662bc0fe45b5a80d5071b442324c71cdcd3c8510db"} Sep 10 23:52:16.092146 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:16.092108 2578 scope.go:117] "RemoveContainer" containerID="ca8c61afd281fffc4da78bb154cdcbdc7f9e4c56d7c4860ad97bca25e45f91c8" Sep 10 23:52:16.093552 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:16.093525 2578 generic.go:356] "Generic (PLEG): container finished" podID="411c7bb1-d006-46c0-882a-aeb3bb9a6904" containerID="170659ce4a9ef6f750a338344fdbb64a93931f5d36cd1d2bbe506c5cf9dbdfb7" exitCode=0 Sep 10 23:52:16.093646 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:16.093570 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt" Sep 10 23:52:16.093646 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:16.093632 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt" event={"ID":"411c7bb1-d006-46c0-882a-aeb3bb9a6904","Type":"ContainerDied","Data":"170659ce4a9ef6f750a338344fdbb64a93931f5d36cd1d2bbe506c5cf9dbdfb7"} Sep 10 23:52:16.093745 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:16.093668 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt" event={"ID":"411c7bb1-d006-46c0-882a-aeb3bb9a6904","Type":"ContainerDied","Data":"44871f5205723d455f69124f99065e08b1c83d09c32bfcbf2c1bece1f85af167"} Sep 10 23:52:16.101051 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:16.101025 2578 scope.go:117] "RemoveContainer" containerID="ca8c61afd281fffc4da78bb154cdcbdc7f9e4c56d7c4860ad97bca25e45f91c8" Sep 10 23:52:16.101341 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:52:16.101323 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca8c61afd281fffc4da78bb154cdcbdc7f9e4c56d7c4860ad97bca25e45f91c8\": container with ID starting with ca8c61afd281fffc4da78bb154cdcbdc7f9e4c56d7c4860ad97bca25e45f91c8 not found: ID does not exist" containerID="ca8c61afd281fffc4da78bb154cdcbdc7f9e4c56d7c4860ad97bca25e45f91c8" Sep 10 23:52:16.101399 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:16.101348 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca8c61afd281fffc4da78bb154cdcbdc7f9e4c56d7c4860ad97bca25e45f91c8"} err="failed to get container status \"ca8c61afd281fffc4da78bb154cdcbdc7f9e4c56d7c4860ad97bca25e45f91c8\": rpc error: code = NotFound desc = could not find container \"ca8c61afd281fffc4da78bb154cdcbdc7f9e4c56d7c4860ad97bca25e45f91c8\": container with ID starting with ca8c61afd281fffc4da78bb154cdcbdc7f9e4c56d7c4860ad97bca25e45f91c8 not found: ID does not exist" Sep 10 23:52:16.101399 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:16.101369 2578 scope.go:117] "RemoveContainer" containerID="170659ce4a9ef6f750a338344fdbb64a93931f5d36cd1d2bbe506c5cf9dbdfb7" Sep 10 23:52:16.109111 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:16.109094 2578 scope.go:117] "RemoveContainer" containerID="170659ce4a9ef6f750a338344fdbb64a93931f5d36cd1d2bbe506c5cf9dbdfb7" Sep 10 23:52:16.109349 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:52:16.109330 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"170659ce4a9ef6f750a338344fdbb64a93931f5d36cd1d2bbe506c5cf9dbdfb7\": container with ID starting with 170659ce4a9ef6f750a338344fdbb64a93931f5d36cd1d2bbe506c5cf9dbdfb7 not found: ID does not exist" containerID="170659ce4a9ef6f750a338344fdbb64a93931f5d36cd1d2bbe506c5cf9dbdfb7" Sep 10 23:52:16.109395 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:16.109356 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"170659ce4a9ef6f750a338344fdbb64a93931f5d36cd1d2bbe506c5cf9dbdfb7"} err="failed to get container status \"170659ce4a9ef6f750a338344fdbb64a93931f5d36cd1d2bbe506c5cf9dbdfb7\": rpc error: code = NotFound desc = could not find container \"170659ce4a9ef6f750a338344fdbb64a93931f5d36cd1d2bbe506c5cf9dbdfb7\": container with ID starting with 170659ce4a9ef6f750a338344fdbb64a93931f5d36cd1d2bbe506c5cf9dbdfb7 not found: ID does not exist" Sep 10 23:52:16.115579 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:16.115558 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2"] Sep 10 23:52:16.119764 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:16.119743 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f9a25dcc-5c66f864f-j9fc2"] Sep 10 23:52:16.129551 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:16.129530 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt"] Sep 10 23:52:16.148961 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:16.136057 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f9a25dcc-767c46788b-mhbdt"] Sep 10 23:52:16.295467 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:16.295367 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk"] Sep 10 23:52:16.295629 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:16.295605 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" podUID="0558f6dd-9393-4fbc-938e-78699eecb86f" containerName="istio-proxy" containerID="cri-o://455107b955ac7753851dd615efb2fc64bb4907005753e3a68ba4ae153c3e6a06" gracePeriod=30 Sep 10 23:52:17.044862 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:17.044820 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="411c7bb1-d006-46c0-882a-aeb3bb9a6904" path="/var/lib/kubelet/pods/411c7bb1-d006-46c0-882a-aeb3bb9a6904/volumes" Sep 10 23:52:17.045362 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:17.045210 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dc0bab2-af48-4152-9df9-189943fb1673" path="/var/lib/kubelet/pods/8dc0bab2-af48-4152-9df9-189943fb1673/volumes" Sep 10 23:52:19.376617 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:19.376574 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv"] Sep 10 23:52:19.377321 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:19.377297 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="411c7bb1-d006-46c0-882a-aeb3bb9a6904" containerName="payload-pre-processing" Sep 10 23:52:19.377321 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:19.377322 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="411c7bb1-d006-46c0-882a-aeb3bb9a6904" containerName="payload-pre-processing" Sep 10 23:52:19.377531 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:19.377345 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aab5922-dba3-4f74-a65d-63b61dc1e138" containerName="payload-pre-processing" Sep 10 23:52:19.377531 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:19.377353 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aab5922-dba3-4f74-a65d-63b61dc1e138" containerName="payload-pre-processing" Sep 10 23:52:19.377531 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:19.377376 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dc0bab2-af48-4152-9df9-189943fb1673" containerName="payload-processing" Sep 10 23:52:19.377531 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:19.377384 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dc0bab2-af48-4152-9df9-189943fb1673" containerName="payload-processing" Sep 10 23:52:19.377712 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:19.377569 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="411c7bb1-d006-46c0-882a-aeb3bb9a6904" containerName="payload-pre-processing" Sep 10 23:52:19.377712 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:19.377580 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dc0bab2-af48-4152-9df9-189943fb1673" containerName="payload-processing" Sep 10 23:52:19.377712 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:19.377596 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5aab5922-dba3-4f74-a65d-63b61dc1e138" containerName="payload-pre-processing" Sep 10 23:52:19.382868 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:19.382845 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv" Sep 10 23:52:19.385870 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:19.385844 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-677d3c-dockercfg-pzdsw\"" Sep 10 23:52:19.386000 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:19.385949 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-677d3c\"" Sep 10 23:52:19.394264 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:19.394237 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv"] Sep 10 23:52:19.521184 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:19.521151 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbe70b8-d8ab-4006-a9e3-aab6628b2984-plugins-config-volume\") pod \"payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv\" (UID: \"4cbe70b8-d8ab-4006-a9e3-aab6628b2984\") " pod="openshift-ingress/payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv" Sep 10 23:52:19.521381 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:19.521220 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h94v\" (UniqueName: \"kubernetes.io/projected/4cbe70b8-d8ab-4006-a9e3-aab6628b2984-kube-api-access-2h94v\") pod \"payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv\" (UID: \"4cbe70b8-d8ab-4006-a9e3-aab6628b2984\") " pod="openshift-ingress/payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv" Sep 10 23:52:19.621959 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:19.621891 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h94v\" (UniqueName: \"kubernetes.io/projected/4cbe70b8-d8ab-4006-a9e3-aab6628b2984-kube-api-access-2h94v\") pod \"payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv\" (UID: \"4cbe70b8-d8ab-4006-a9e3-aab6628b2984\") " pod="openshift-ingress/payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv" Sep 10 23:52:19.622144 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:19.622044 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbe70b8-d8ab-4006-a9e3-aab6628b2984-plugins-config-volume\") pod \"payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv\" (UID: \"4cbe70b8-d8ab-4006-a9e3-aab6628b2984\") " pod="openshift-ingress/payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv" Sep 10 23:52:19.622719 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:19.622699 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbe70b8-d8ab-4006-a9e3-aab6628b2984-plugins-config-volume\") pod \"payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv\" (UID: \"4cbe70b8-d8ab-4006-a9e3-aab6628b2984\") " pod="openshift-ingress/payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv" Sep 10 23:52:19.630230 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:19.630167 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h94v\" (UniqueName: \"kubernetes.io/projected/4cbe70b8-d8ab-4006-a9e3-aab6628b2984-kube-api-access-2h94v\") pod \"payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv\" (UID: \"4cbe70b8-d8ab-4006-a9e3-aab6628b2984\") " pod="openshift-ingress/payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv" Sep 10 23:52:19.694050 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:19.694013 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv" Sep 10 23:52:20.013553 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:20.013457 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv"] Sep 10 23:52:20.015982 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:52:20.015955 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cbe70b8_d8ab_4006_a9e3_aab6628b2984.slice/crio-9c18147c374b290522b0dcea047a49e3851ef7d1f92fea38fc25b341d98bc5be WatchSource:0}: Error finding container 9c18147c374b290522b0dcea047a49e3851ef7d1f92fea38fc25b341d98bc5be: Status 404 returned error can't find the container with id 9c18147c374b290522b0dcea047a49e3851ef7d1f92fea38fc25b341d98bc5be Sep 10 23:52:20.116209 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:20.116174 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv" event={"ID":"4cbe70b8-d8ab-4006-a9e3-aab6628b2984","Type":"ContainerStarted","Data":"1d3293696a1b6f2bda01522de39574f15a8e6c20a31c427ab5b4cc6b33eb8928"} Sep 10 23:52:20.116359 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:20.116224 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv" event={"ID":"4cbe70b8-d8ab-4006-a9e3-aab6628b2984","Type":"ContainerStarted","Data":"9c18147c374b290522b0dcea047a49e3851ef7d1f92fea38fc25b341d98bc5be"} Sep 10 23:52:20.116359 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:20.116322 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv" Sep 10 23:52:20.153280 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:20.153205 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv" podStartSLOduration=1.153179198 podStartE2EDuration="1.153179198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:52:20.150739034 +0000 UTC m=+955.618213371" watchObservedRunningTime="2026-09-10 23:52:20.153179198 +0000 UTC m=+955.620653524" Sep 10 23:52:22.208159 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.208127 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2"] Sep 10 23:52:22.211447 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.211428 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2" Sep 10 23:52:22.214120 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.214100 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-15a3a094-dockercfg-g4jp7\"" Sep 10 23:52:22.214216 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.214104 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-15a3a094\"" Sep 10 23:52:22.224339 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.224316 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2"] Sep 10 23:52:22.345127 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.345081 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3060cd5-86d5-49bc-a4ba-8c36af87f82a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2\" (UID: \"b3060cd5-86d5-49bc-a4ba-8c36af87f82a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2" Sep 10 23:52:22.345293 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.345184 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gnsp\" (UniqueName: \"kubernetes.io/projected/b3060cd5-86d5-49bc-a4ba-8c36af87f82a-kube-api-access-7gnsp\") pod \"payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2\" (UID: \"b3060cd5-86d5-49bc-a4ba-8c36af87f82a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2" Sep 10 23:52:22.446033 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.446005 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gnsp\" (UniqueName: \"kubernetes.io/projected/b3060cd5-86d5-49bc-a4ba-8c36af87f82a-kube-api-access-7gnsp\") pod \"payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2\" (UID: \"b3060cd5-86d5-49bc-a4ba-8c36af87f82a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2" Sep 10 23:52:22.446142 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.446071 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3060cd5-86d5-49bc-a4ba-8c36af87f82a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2\" (UID: \"b3060cd5-86d5-49bc-a4ba-8c36af87f82a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2" Sep 10 23:52:22.446699 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.446680 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3060cd5-86d5-49bc-a4ba-8c36af87f82a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2\" (UID: \"b3060cd5-86d5-49bc-a4ba-8c36af87f82a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2" Sep 10 23:52:22.456614 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.456565 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:52:22.466133 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.466112 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gnsp\" (UniqueName: \"kubernetes.io/projected/b3060cd5-86d5-49bc-a4ba-8c36af87f82a-kube-api-access-7gnsp\") pod \"payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2\" (UID: \"b3060cd5-86d5-49bc-a4ba-8c36af87f82a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2" Sep 10 23:52:22.547024 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.546992 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0558f6dd-9393-4fbc-938e-78699eecb86f-istiod-ca-cert\") pod \"0558f6dd-9393-4fbc-938e-78699eecb86f\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " Sep 10 23:52:22.547181 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.547029 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0558f6dd-9393-4fbc-938e-78699eecb86f-istio-podinfo\") pod \"0558f6dd-9393-4fbc-938e-78699eecb86f\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " Sep 10 23:52:22.547181 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.547055 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-workload-socket\") pod \"0558f6dd-9393-4fbc-938e-78699eecb86f\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " Sep 10 23:52:22.547181 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.547081 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-istio-envoy\") pod \"0558f6dd-9393-4fbc-938e-78699eecb86f\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " Sep 10 23:52:22.547181 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.547112 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-workload-certs\") pod \"0558f6dd-9393-4fbc-938e-78699eecb86f\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " Sep 10 23:52:22.547181 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.547142 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x827f\" (UniqueName: \"kubernetes.io/projected/0558f6dd-9393-4fbc-938e-78699eecb86f-kube-api-access-x827f\") pod \"0558f6dd-9393-4fbc-938e-78699eecb86f\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " Sep 10 23:52:22.547465 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.547243 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0558f6dd-9393-4fbc-938e-78699eecb86f-istio-token\") pod \"0558f6dd-9393-4fbc-938e-78699eecb86f\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " Sep 10 23:52:22.547465 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.547284 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-istio-data\") pod \"0558f6dd-9393-4fbc-938e-78699eecb86f\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " Sep 10 23:52:22.547465 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.547310 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-credential-socket\") pod \"0558f6dd-9393-4fbc-938e-78699eecb86f\" (UID: \"0558f6dd-9393-4fbc-938e-78699eecb86f\") " Sep 10 23:52:22.547613 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.547484 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0558f6dd-9393-4fbc-938e-78699eecb86f" (UID: "0558f6dd-9393-4fbc-938e-78699eecb86f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:52:22.547613 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.547596 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0558f6dd-9393-4fbc-938e-78699eecb86f" (UID: "0558f6dd-9393-4fbc-938e-78699eecb86f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:52:22.547706 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.547511 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0558f6dd-9393-4fbc-938e-78699eecb86f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0558f6dd-9393-4fbc-938e-78699eecb86f" (UID: "0558f6dd-9393-4fbc-938e-78699eecb86f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:52:22.547706 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.547661 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-workload-certs\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:52:22.547800 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.547725 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0558f6dd-9393-4fbc-938e-78699eecb86f" (UID: "0558f6dd-9393-4fbc-938e-78699eecb86f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:52:22.547859 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.547801 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-istio-data" (OuterVolumeSpecName: "istio-data") pod "0558f6dd-9393-4fbc-938e-78699eecb86f" (UID: "0558f6dd-9393-4fbc-938e-78699eecb86f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:52:22.549400 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.549366 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0558f6dd-9393-4fbc-938e-78699eecb86f-kube-api-access-x827f" (OuterVolumeSpecName: "kube-api-access-x827f") pod "0558f6dd-9393-4fbc-938e-78699eecb86f" (UID: "0558f6dd-9393-4fbc-938e-78699eecb86f"). InnerVolumeSpecName "kube-api-access-x827f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:52:22.549509 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.549412 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0558f6dd-9393-4fbc-938e-78699eecb86f-istio-token" (OuterVolumeSpecName: "istio-token") pod "0558f6dd-9393-4fbc-938e-78699eecb86f" (UID: "0558f6dd-9393-4fbc-938e-78699eecb86f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:52:22.549509 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.549438 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0558f6dd-9393-4fbc-938e-78699eecb86f" (UID: "0558f6dd-9393-4fbc-938e-78699eecb86f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:52:22.549509 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.549476 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0558f6dd-9393-4fbc-938e-78699eecb86f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0558f6dd-9393-4fbc-938e-78699eecb86f" (UID: "0558f6dd-9393-4fbc-938e-78699eecb86f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 23:52:22.564670 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.564650 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2" Sep 10 23:52:22.648595 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.648565 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0558f6dd-9393-4fbc-938e-78699eecb86f-istio-token\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:52:22.648595 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.648595 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-istio-data\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:52:22.648794 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.648607 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-credential-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:52:22.648794 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.648621 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0558f6dd-9393-4fbc-938e-78699eecb86f-istiod-ca-cert\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:52:22.648794 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.648635 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0558f6dd-9393-4fbc-938e-78699eecb86f-istio-podinfo\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:52:22.648794 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.648645 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-workload-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:52:22.648794 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.648656 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0558f6dd-9393-4fbc-938e-78699eecb86f-istio-envoy\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:52:22.648794 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.648668 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x827f\" (UniqueName: \"kubernetes.io/projected/0558f6dd-9393-4fbc-938e-78699eecb86f-kube-api-access-x827f\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:52:22.843283 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:22.843256 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2"] Sep 10 23:52:22.845210 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:52:22.845183 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3060cd5_86d5_49bc_a4ba_8c36af87f82a.slice/crio-01969db09d7d3dc97ee2bf6116749a612118b345cd1fb9bfac31080909b8c14c WatchSource:0}: Error finding container 01969db09d7d3dc97ee2bf6116749a612118b345cd1fb9bfac31080909b8c14c: Status 404 returned error can't find the container with id 01969db09d7d3dc97ee2bf6116749a612118b345cd1fb9bfac31080909b8c14c Sep 10 23:52:23.132517 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:23.132481 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2" event={"ID":"b3060cd5-86d5-49bc-a4ba-8c36af87f82a","Type":"ContainerStarted","Data":"771291d46e7191117fa3af98613d52330ff80f17f72041bddf097ea1eb676d3f"} Sep 10 23:52:23.132702 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:23.132543 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2" event={"ID":"b3060cd5-86d5-49bc-a4ba-8c36af87f82a","Type":"ContainerStarted","Data":"01969db09d7d3dc97ee2bf6116749a612118b345cd1fb9bfac31080909b8c14c"} Sep 10 23:52:23.132702 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:23.132601 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2" Sep 10 23:52:23.134222 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:23.134200 2578 generic.go:356] "Generic (PLEG): container finished" podID="0558f6dd-9393-4fbc-938e-78699eecb86f" containerID="455107b955ac7753851dd615efb2fc64bb4907005753e3a68ba4ae153c3e6a06" exitCode=0 Sep 10 23:52:23.134331 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:23.134225 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" event={"ID":"0558f6dd-9393-4fbc-938e-78699eecb86f","Type":"ContainerDied","Data":"455107b955ac7753851dd615efb2fc64bb4907005753e3a68ba4ae153c3e6a06"} Sep 10 23:52:23.134331 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:23.134254 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" Sep 10 23:52:23.134331 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:23.134279 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk" event={"ID":"0558f6dd-9393-4fbc-938e-78699eecb86f","Type":"ContainerDied","Data":"236e0d5cd2704486ee1e52388d4861c63d4395054e2535f6c2e13d3b648845ee"} Sep 10 23:52:23.134331 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:23.134311 2578 scope.go:117] "RemoveContainer" containerID="455107b955ac7753851dd615efb2fc64bb4907005753e3a68ba4ae153c3e6a06" Sep 10 23:52:23.143069 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:23.143047 2578 scope.go:117] "RemoveContainer" containerID="be9a6f6d54cc9948f57281448b1e57259386d5e77e57380917eee86065a4b69d" Sep 10 23:52:23.151111 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:23.151070 2578 scope.go:117] "RemoveContainer" containerID="455107b955ac7753851dd615efb2fc64bb4907005753e3a68ba4ae153c3e6a06" Sep 10 23:52:23.156000 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:23.154643 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2" podStartSLOduration=1.154626731 podStartE2EDuration="1.154626731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:52:23.150621671 +0000 UTC m=+958.618096000" watchObservedRunningTime="2026-09-10 23:52:23.154626731 +0000 UTC m=+958.622101057" Sep 10 23:52:23.156000 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:52:23.155302 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"455107b955ac7753851dd615efb2fc64bb4907005753e3a68ba4ae153c3e6a06\": container with ID starting with 455107b955ac7753851dd615efb2fc64bb4907005753e3a68ba4ae153c3e6a06 not found: ID does not exist" containerID="455107b955ac7753851dd615efb2fc64bb4907005753e3a68ba4ae153c3e6a06" Sep 10 23:52:23.156000 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:23.155347 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"455107b955ac7753851dd615efb2fc64bb4907005753e3a68ba4ae153c3e6a06"} err="failed to get container status \"455107b955ac7753851dd615efb2fc64bb4907005753e3a68ba4ae153c3e6a06\": rpc error: code = NotFound desc = could not find container \"455107b955ac7753851dd615efb2fc64bb4907005753e3a68ba4ae153c3e6a06\": container with ID starting with 455107b955ac7753851dd615efb2fc64bb4907005753e3a68ba4ae153c3e6a06 not found: ID does not exist" Sep 10 23:52:23.156000 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:23.155368 2578 scope.go:117] "RemoveContainer" containerID="be9a6f6d54cc9948f57281448b1e57259386d5e77e57380917eee86065a4b69d" Sep 10 23:52:23.156000 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:52:23.155664 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be9a6f6d54cc9948f57281448b1e57259386d5e77e57380917eee86065a4b69d\": container with ID starting with be9a6f6d54cc9948f57281448b1e57259386d5e77e57380917eee86065a4b69d not found: ID does not exist" containerID="be9a6f6d54cc9948f57281448b1e57259386d5e77e57380917eee86065a4b69d" Sep 10 23:52:23.156000 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:23.155698 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9a6f6d54cc9948f57281448b1e57259386d5e77e57380917eee86065a4b69d"} err="failed to get container status \"be9a6f6d54cc9948f57281448b1e57259386d5e77e57380917eee86065a4b69d\": rpc error: code = NotFound desc = could not find container \"be9a6f6d54cc9948f57281448b1e57259386d5e77e57380917eee86065a4b69d\": container with ID starting with be9a6f6d54cc9948f57281448b1e57259386d5e77e57380917eee86065a4b69d not found: ID does not exist" Sep 10 23:52:23.166301 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:23.166279 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk"] Sep 10 23:52:23.171149 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:23.171129 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f9a25dcc-openshift-default-fbfbb4498-kcfvk"] Sep 10 23:52:25.043979 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:25.043927 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0558f6dd-9393-4fbc-938e-78699eecb86f" path="/var/lib/kubelet/pods/0558f6dd-9393-4fbc-938e-78699eecb86f/volumes" Sep 10 23:52:31.122955 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:31.122913 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv" Sep 10 23:52:34.142922 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:34.142892 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2" Sep 10 23:52:52.069387 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:52.069295 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2"] Sep 10 23:52:52.069999 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:52.069560 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2" podUID="b3060cd5-86d5-49bc-a4ba-8c36af87f82a" containerName="payload-pre-processing" containerID="cri-o://771291d46e7191117fa3af98613d52330ff80f17f72041bddf097ea1eb676d3f" gracePeriod=30 Sep 10 23:52:52.277765 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:52.277726 2578 generic.go:356] "Generic (PLEG): container finished" podID="b3060cd5-86d5-49bc-a4ba-8c36af87f82a" containerID="771291d46e7191117fa3af98613d52330ff80f17f72041bddf097ea1eb676d3f" exitCode=0 Sep 10 23:52:52.277974 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:52.277780 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2" event={"ID":"b3060cd5-86d5-49bc-a4ba-8c36af87f82a","Type":"ContainerDied","Data":"771291d46e7191117fa3af98613d52330ff80f17f72041bddf097ea1eb676d3f"} Sep 10 23:52:52.322386 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:52.322329 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2" Sep 10 23:52:52.405691 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:52.405659 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gnsp\" (UniqueName: \"kubernetes.io/projected/b3060cd5-86d5-49bc-a4ba-8c36af87f82a-kube-api-access-7gnsp\") pod \"b3060cd5-86d5-49bc-a4ba-8c36af87f82a\" (UID: \"b3060cd5-86d5-49bc-a4ba-8c36af87f82a\") " Sep 10 23:52:52.405876 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:52.405744 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3060cd5-86d5-49bc-a4ba-8c36af87f82a-plugins-config-volume\") pod \"b3060cd5-86d5-49bc-a4ba-8c36af87f82a\" (UID: \"b3060cd5-86d5-49bc-a4ba-8c36af87f82a\") " Sep 10 23:52:52.406208 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:52.406180 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3060cd5-86d5-49bc-a4ba-8c36af87f82a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b3060cd5-86d5-49bc-a4ba-8c36af87f82a" (UID: "b3060cd5-86d5-49bc-a4ba-8c36af87f82a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:52:52.408633 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:52.408608 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3060cd5-86d5-49bc-a4ba-8c36af87f82a-kube-api-access-7gnsp" (OuterVolumeSpecName: "kube-api-access-7gnsp") pod "b3060cd5-86d5-49bc-a4ba-8c36af87f82a" (UID: "b3060cd5-86d5-49bc-a4ba-8c36af87f82a"). InnerVolumeSpecName "kube-api-access-7gnsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:52:52.507122 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:52.507094 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3060cd5-86d5-49bc-a4ba-8c36af87f82a-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:52:52.507122 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:52.507120 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gnsp\" (UniqueName: \"kubernetes.io/projected/b3060cd5-86d5-49bc-a4ba-8c36af87f82a-kube-api-access-7gnsp\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:52:53.284737 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:53.284704 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2" event={"ID":"b3060cd5-86d5-49bc-a4ba-8c36af87f82a","Type":"ContainerDied","Data":"01969db09d7d3dc97ee2bf6116749a612118b345cd1fb9bfac31080909b8c14c"} Sep 10 23:52:53.284737 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:53.284738 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2" Sep 10 23:52:53.285229 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:53.284767 2578 scope.go:117] "RemoveContainer" containerID="771291d46e7191117fa3af98613d52330ff80f17f72041bddf097ea1eb676d3f" Sep 10 23:52:53.310043 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:53.309987 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2"] Sep 10 23:52:53.316743 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:53.316718 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-15a3a094-7bbb9dd6b-xh6h2"] Sep 10 23:52:55.044516 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:52:55.044479 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3060cd5-86d5-49bc-a4ba-8c36af87f82a" path="/var/lib/kubelet/pods/b3060cd5-86d5-49bc-a4ba-8c36af87f82a/volumes" Sep 10 23:53:46.115536 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:46.115500 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv"] Sep 10 23:53:46.116059 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:46.115711 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv" podUID="4cbe70b8-d8ab-4006-a9e3-aab6628b2984" containerName="payload-processing" containerID="cri-o://1d3293696a1b6f2bda01522de39574f15a8e6c20a31c427ab5b4cc6b33eb8928" gracePeriod=30 Sep 10 23:53:46.371240 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:46.371187 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv" Sep 10 23:53:46.464805 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:46.464772 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbe70b8-d8ab-4006-a9e3-aab6628b2984-plugins-config-volume\") pod \"4cbe70b8-d8ab-4006-a9e3-aab6628b2984\" (UID: \"4cbe70b8-d8ab-4006-a9e3-aab6628b2984\") " Sep 10 23:53:46.464980 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:46.464852 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h94v\" (UniqueName: \"kubernetes.io/projected/4cbe70b8-d8ab-4006-a9e3-aab6628b2984-kube-api-access-2h94v\") pod \"4cbe70b8-d8ab-4006-a9e3-aab6628b2984\" (UID: \"4cbe70b8-d8ab-4006-a9e3-aab6628b2984\") " Sep 10 23:53:46.465139 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:46.465115 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cbe70b8-d8ab-4006-a9e3-aab6628b2984-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4cbe70b8-d8ab-4006-a9e3-aab6628b2984" (UID: "4cbe70b8-d8ab-4006-a9e3-aab6628b2984"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:53:46.466841 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:46.466813 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbe70b8-d8ab-4006-a9e3-aab6628b2984-kube-api-access-2h94v" (OuterVolumeSpecName: "kube-api-access-2h94v") pod "4cbe70b8-d8ab-4006-a9e3-aab6628b2984" (UID: "4cbe70b8-d8ab-4006-a9e3-aab6628b2984"). InnerVolumeSpecName "kube-api-access-2h94v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:53:46.529623 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:46.529589 2578 generic.go:356] "Generic (PLEG): container finished" podID="4cbe70b8-d8ab-4006-a9e3-aab6628b2984" containerID="1d3293696a1b6f2bda01522de39574f15a8e6c20a31c427ab5b4cc6b33eb8928" exitCode=0 Sep 10 23:53:46.529795 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:46.529628 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv" Sep 10 23:53:46.529795 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:46.529683 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv" event={"ID":"4cbe70b8-d8ab-4006-a9e3-aab6628b2984","Type":"ContainerDied","Data":"1d3293696a1b6f2bda01522de39574f15a8e6c20a31c427ab5b4cc6b33eb8928"} Sep 10 23:53:46.529795 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:46.529727 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv" event={"ID":"4cbe70b8-d8ab-4006-a9e3-aab6628b2984","Type":"ContainerDied","Data":"9c18147c374b290522b0dcea047a49e3851ef7d1f92fea38fc25b341d98bc5be"} Sep 10 23:53:46.529795 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:46.529749 2578 scope.go:117] "RemoveContainer" containerID="1d3293696a1b6f2bda01522de39574f15a8e6c20a31c427ab5b4cc6b33eb8928" Sep 10 23:53:46.538101 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:46.538084 2578 scope.go:117] "RemoveContainer" containerID="1d3293696a1b6f2bda01522de39574f15a8e6c20a31c427ab5b4cc6b33eb8928" Sep 10 23:53:46.538339 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:53:46.538321 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d3293696a1b6f2bda01522de39574f15a8e6c20a31c427ab5b4cc6b33eb8928\": container with ID starting with 1d3293696a1b6f2bda01522de39574f15a8e6c20a31c427ab5b4cc6b33eb8928 not found: ID does not exist" containerID="1d3293696a1b6f2bda01522de39574f15a8e6c20a31c427ab5b4cc6b33eb8928" Sep 10 23:53:46.538403 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:46.538352 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d3293696a1b6f2bda01522de39574f15a8e6c20a31c427ab5b4cc6b33eb8928"} err="failed to get container status \"1d3293696a1b6f2bda01522de39574f15a8e6c20a31c427ab5b4cc6b33eb8928\": rpc error: code = NotFound desc = could not find container \"1d3293696a1b6f2bda01522de39574f15a8e6c20a31c427ab5b4cc6b33eb8928\": container with ID starting with 1d3293696a1b6f2bda01522de39574f15a8e6c20a31c427ab5b4cc6b33eb8928 not found: ID does not exist" Sep 10 23:53:46.551291 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:46.551269 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv"] Sep 10 23:53:46.559035 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:46.559004 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-677d3c-6679689cbf-w6qlv"] Sep 10 23:53:46.565779 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:46.565759 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbe70b8-d8ab-4006-a9e3-aab6628b2984-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:53:46.565779 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:46.565780 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h94v\" (UniqueName: \"kubernetes.io/projected/4cbe70b8-d8ab-4006-a9e3-aab6628b2984-kube-api-access-2h94v\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:53:47.044392 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:47.044358 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cbe70b8-d8ab-4006-a9e3-aab6628b2984" path="/var/lib/kubelet/pods/4cbe70b8-d8ab-4006-a9e3-aab6628b2984/volumes" Sep 10 23:53:53.605303 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.605261 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q"] Sep 10 23:53:53.605862 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.605803 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0558f6dd-9393-4fbc-938e-78699eecb86f" containerName="istio-proxy" Sep 10 23:53:53.605862 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.605826 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0558f6dd-9393-4fbc-938e-78699eecb86f" containerName="istio-proxy" Sep 10 23:53:53.605862 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.605851 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cbe70b8-d8ab-4006-a9e3-aab6628b2984" containerName="payload-processing" Sep 10 23:53:53.605862 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.605859 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cbe70b8-d8ab-4006-a9e3-aab6628b2984" containerName="payload-processing" Sep 10 23:53:53.606112 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.605876 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3060cd5-86d5-49bc-a4ba-8c36af87f82a" containerName="payload-pre-processing" Sep 10 23:53:53.606112 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.605885 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3060cd5-86d5-49bc-a4ba-8c36af87f82a" containerName="payload-pre-processing" Sep 10 23:53:53.606112 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.606003 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0558f6dd-9393-4fbc-938e-78699eecb86f" containerName="istio-proxy" Sep 10 23:53:53.606112 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.606018 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0558f6dd-9393-4fbc-938e-78699eecb86f" containerName="istio-proxy" Sep 10 23:53:53.606112 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.606028 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3060cd5-86d5-49bc-a4ba-8c36af87f82a" containerName="payload-pre-processing" Sep 10 23:53:53.606112 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.606040 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cbe70b8-d8ab-4006-a9e3-aab6628b2984" containerName="payload-processing" Sep 10 23:53:53.608855 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.608834 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.612194 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.612173 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-df1107-openshift-default-dockercfg-b94fx\"" Sep 10 23:53:53.631688 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.631658 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q"] Sep 10 23:53:53.733122 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.733091 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istio-token\") pod \"e2e-noten-df1107-openshift-default-86bdcb7474-dp62q\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.733310 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.733134 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-workload-certs\") pod \"e2e-noten-df1107-openshift-default-86bdcb7474-dp62q\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.733310 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.733157 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istio-envoy\") pod \"e2e-noten-df1107-openshift-default-86bdcb7474-dp62q\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.733310 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.733272 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istio-podinfo\") pod \"e2e-noten-df1107-openshift-default-86bdcb7474-dp62q\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.733460 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.733311 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-credential-socket\") pod \"e2e-noten-df1107-openshift-default-86bdcb7474-dp62q\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.733460 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.733336 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istio-data\") pod \"e2e-noten-df1107-openshift-default-86bdcb7474-dp62q\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.733460 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.733383 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-workload-socket\") pod \"e2e-noten-df1107-openshift-default-86bdcb7474-dp62q\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.733460 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.733414 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2t8t\" (UniqueName: \"kubernetes.io/projected/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-kube-api-access-x2t8t\") pod \"e2e-noten-df1107-openshift-default-86bdcb7474-dp62q\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.733460 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.733452 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istiod-ca-cert\") pod \"e2e-noten-df1107-openshift-default-86bdcb7474-dp62q\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.834493 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.834456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istio-data\") pod \"e2e-noten-df1107-openshift-default-86bdcb7474-dp62q\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.834493 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.834493 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-workload-socket\") pod \"e2e-noten-df1107-openshift-default-86bdcb7474-dp62q\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.834735 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.834515 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2t8t\" (UniqueName: \"kubernetes.io/projected/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-kube-api-access-x2t8t\") pod \"e2e-noten-df1107-openshift-default-86bdcb7474-dp62q\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.834735 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.834539 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istiod-ca-cert\") pod \"e2e-noten-df1107-openshift-default-86bdcb7474-dp62q\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.834735 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.834562 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istio-token\") pod \"e2e-noten-df1107-openshift-default-86bdcb7474-dp62q\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.834735 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.834593 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-workload-certs\") pod \"e2e-noten-df1107-openshift-default-86bdcb7474-dp62q\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.834735 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.834616 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istio-envoy\") pod \"e2e-noten-df1107-openshift-default-86bdcb7474-dp62q\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.834735 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.834648 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istio-podinfo\") pod \"e2e-noten-df1107-openshift-default-86bdcb7474-dp62q\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.834735 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.834692 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-credential-socket\") pod \"e2e-noten-df1107-openshift-default-86bdcb7474-dp62q\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.835125 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.834925 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istio-data\") pod \"e2e-noten-df1107-openshift-default-86bdcb7474-dp62q\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.835125 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.835060 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-workload-certs\") pod \"e2e-noten-df1107-openshift-default-86bdcb7474-dp62q\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.835206 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.835140 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-credential-socket\") pod \"e2e-noten-df1107-openshift-default-86bdcb7474-dp62q\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.835241 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.835210 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-workload-socket\") pod \"e2e-noten-df1107-openshift-default-86bdcb7474-dp62q\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.835318 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.835299 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istiod-ca-cert\") pod \"e2e-noten-df1107-openshift-default-86bdcb7474-dp62q\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.837076 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.837059 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istio-envoy\") pod \"e2e-noten-df1107-openshift-default-86bdcb7474-dp62q\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.837129 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.837115 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istio-podinfo\") pod \"e2e-noten-df1107-openshift-default-86bdcb7474-dp62q\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.843333 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.843282 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istio-token\") pod \"e2e-noten-df1107-openshift-default-86bdcb7474-dp62q\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.844124 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.844099 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2t8t\" (UniqueName: \"kubernetes.io/projected/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-kube-api-access-x2t8t\") pod \"e2e-noten-df1107-openshift-default-86bdcb7474-dp62q\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:53.919956 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:53.919904 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:54.195503 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:54.195474 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q"] Sep 10 23:53:54.197357 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:53:54.197330 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ced2b3c_effb_40be_b02b_9a7aff37a4b0.slice/crio-6a4b325838f6a1153aeaeee6d2c4a33ebd4cf56f71109fe11212f4cb125f884d WatchSource:0}: Error finding container 6a4b325838f6a1153aeaeee6d2c4a33ebd4cf56f71109fe11212f4cb125f884d: Status 404 returned error can't find the container with id 6a4b325838f6a1153aeaeee6d2c4a33ebd4cf56f71109fe11212f4cb125f884d Sep 10 23:53:54.199522 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:54.199486 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:53:54.199601 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:54.199564 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:53:54.199643 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:54.199606 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:53:54.569243 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:54.569165 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" event={"ID":"7ced2b3c-effb-40be-b02b-9a7aff37a4b0","Type":"ContainerStarted","Data":"0256e315fb1f9005761410cdd6cf232ffc53796abccf07da8154106ce10c17b5"} Sep 10 23:53:54.569243 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:54.569216 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" event={"ID":"7ced2b3c-effb-40be-b02b-9a7aff37a4b0","Type":"ContainerStarted","Data":"6a4b325838f6a1153aeaeee6d2c4a33ebd4cf56f71109fe11212f4cb125f884d"} Sep 10 23:53:54.591414 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:54.591361 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" podStartSLOduration=1.591342278 podStartE2EDuration="1.591342278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:53:54.59014702 +0000 UTC m=+1050.057621346" watchObservedRunningTime="2026-09-10 23:53:54.591342278 +0000 UTC m=+1050.058816602" Sep 10 23:53:54.920712 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:54.920676 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:55.926033 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:55.926003 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:56.578793 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:56.578764 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:53:56.579707 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:53:56.579684 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:54:22.824522 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:22.824435 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q"] Sep 10 23:54:22.824891 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:22.824726 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" podUID="7ced2b3c-effb-40be-b02b-9a7aff37a4b0" containerName="istio-proxy" containerID="cri-o://0256e315fb1f9005761410cdd6cf232ffc53796abccf07da8154106ce10c17b5" gracePeriod=30 Sep 10 23:54:26.581636 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:26.581601 2578 patch_prober.go:28] interesting pod/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Sep 10 23:54:26.582082 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:26.581655 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" podUID="7ced2b3c-effb-40be-b02b-9a7aff37a4b0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 23:54:28.080214 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.080187 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:54:28.127220 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.127188 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-workload-certs\") pod \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " Sep 10 23:54:28.127382 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.127225 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istio-envoy\") pod \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " Sep 10 23:54:28.127382 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.127277 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2t8t\" (UniqueName: \"kubernetes.io/projected/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-kube-api-access-x2t8t\") pod \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " Sep 10 23:54:28.127382 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.127311 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istiod-ca-cert\") pod \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " Sep 10 23:54:28.127382 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.127360 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istio-data\") pod \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " Sep 10 23:54:28.127601 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.127397 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-workload-socket\") pod \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " Sep 10 23:54:28.127601 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.127451 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istio-podinfo\") pod \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " Sep 10 23:54:28.127601 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.127474 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istio-token\") pod \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " Sep 10 23:54:28.127601 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.127484 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7ced2b3c-effb-40be-b02b-9a7aff37a4b0" (UID: "7ced2b3c-effb-40be-b02b-9a7aff37a4b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:54:28.127601 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.127515 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-credential-socket\") pod \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\" (UID: \"7ced2b3c-effb-40be-b02b-9a7aff37a4b0\") " Sep 10 23:54:28.127859 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.127671 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istio-data" (OuterVolumeSpecName: "istio-data") pod "7ced2b3c-effb-40be-b02b-9a7aff37a4b0" (UID: "7ced2b3c-effb-40be-b02b-9a7aff37a4b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:54:28.127859 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.127706 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7ced2b3c-effb-40be-b02b-9a7aff37a4b0" (UID: "7ced2b3c-effb-40be-b02b-9a7aff37a4b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:54:28.127859 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.127760 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7ced2b3c-effb-40be-b02b-9a7aff37a4b0" (UID: "7ced2b3c-effb-40be-b02b-9a7aff37a4b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:54:28.128058 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.127906 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7ced2b3c-effb-40be-b02b-9a7aff37a4b0" (UID: "7ced2b3c-effb-40be-b02b-9a7aff37a4b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:54:28.128193 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.128172 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-credential-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:54:28.128262 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.128199 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-workload-certs\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:54:28.128262 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.128214 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istiod-ca-cert\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:54:28.128262 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.128226 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istio-data\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:54:28.128262 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.128240 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-workload-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:54:28.129487 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.129461 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7ced2b3c-effb-40be-b02b-9a7aff37a4b0" (UID: "7ced2b3c-effb-40be-b02b-9a7aff37a4b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:54:28.129689 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.129665 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7ced2b3c-effb-40be-b02b-9a7aff37a4b0" (UID: "7ced2b3c-effb-40be-b02b-9a7aff37a4b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 23:54:28.129896 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.129870 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-kube-api-access-x2t8t" (OuterVolumeSpecName: "kube-api-access-x2t8t") pod "7ced2b3c-effb-40be-b02b-9a7aff37a4b0" (UID: "7ced2b3c-effb-40be-b02b-9a7aff37a4b0"). InnerVolumeSpecName "kube-api-access-x2t8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:54:28.130016 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.129954 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istio-token" (OuterVolumeSpecName: "istio-token") pod "7ced2b3c-effb-40be-b02b-9a7aff37a4b0" (UID: "7ced2b3c-effb-40be-b02b-9a7aff37a4b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:54:28.229486 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.229458 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istio-podinfo\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:54:28.229486 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.229483 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istio-token\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:54:28.229486 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.229493 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-istio-envoy\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:54:28.229707 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.229503 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2t8t\" (UniqueName: \"kubernetes.io/projected/7ced2b3c-effb-40be-b02b-9a7aff37a4b0-kube-api-access-x2t8t\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:54:28.729785 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.729752 2578 generic.go:356] "Generic (PLEG): container finished" podID="7ced2b3c-effb-40be-b02b-9a7aff37a4b0" containerID="0256e315fb1f9005761410cdd6cf232ffc53796abccf07da8154106ce10c17b5" exitCode=0 Sep 10 23:54:28.730005 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.729809 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" Sep 10 23:54:28.730005 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.729837 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" event={"ID":"7ced2b3c-effb-40be-b02b-9a7aff37a4b0","Type":"ContainerDied","Data":"0256e315fb1f9005761410cdd6cf232ffc53796abccf07da8154106ce10c17b5"} Sep 10 23:54:28.730005 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.729928 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q" event={"ID":"7ced2b3c-effb-40be-b02b-9a7aff37a4b0","Type":"ContainerDied","Data":"6a4b325838f6a1153aeaeee6d2c4a33ebd4cf56f71109fe11212f4cb125f884d"} Sep 10 23:54:28.730005 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.729961 2578 scope.go:117] "RemoveContainer" containerID="0256e315fb1f9005761410cdd6cf232ffc53796abccf07da8154106ce10c17b5" Sep 10 23:54:28.741121 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.741101 2578 scope.go:117] "RemoveContainer" containerID="0256e315fb1f9005761410cdd6cf232ffc53796abccf07da8154106ce10c17b5" Sep 10 23:54:28.741502 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:54:28.741483 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0256e315fb1f9005761410cdd6cf232ffc53796abccf07da8154106ce10c17b5\": container with ID starting with 0256e315fb1f9005761410cdd6cf232ffc53796abccf07da8154106ce10c17b5 not found: ID does not exist" containerID="0256e315fb1f9005761410cdd6cf232ffc53796abccf07da8154106ce10c17b5" Sep 10 23:54:28.741559 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.741511 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0256e315fb1f9005761410cdd6cf232ffc53796abccf07da8154106ce10c17b5"} err="failed to get container status \"0256e315fb1f9005761410cdd6cf232ffc53796abccf07da8154106ce10c17b5\": rpc error: code = NotFound desc = could not find container \"0256e315fb1f9005761410cdd6cf232ffc53796abccf07da8154106ce10c17b5\": container with ID starting with 0256e315fb1f9005761410cdd6cf232ffc53796abccf07da8154106ce10c17b5 not found: ID does not exist" Sep 10 23:54:28.755392 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.755365 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q"] Sep 10 23:54:28.763431 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:28.763411 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-df1107-openshift-default-86bdcb7474-dp62q"] Sep 10 23:54:29.044463 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:29.044390 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ced2b3c-effb-40be-b02b-9a7aff37a4b0" path="/var/lib/kubelet/pods/7ced2b3c-effb-40be-b02b-9a7aff37a4b0/volumes" Sep 10 23:54:34.402255 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.402218 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592"] Sep 10 23:54:34.402749 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.402729 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ced2b3c-effb-40be-b02b-9a7aff37a4b0" containerName="istio-proxy" Sep 10 23:54:34.402825 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.402753 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ced2b3c-effb-40be-b02b-9a7aff37a4b0" containerName="istio-proxy" Sep 10 23:54:34.402825 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.402789 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0558f6dd-9393-4fbc-938e-78699eecb86f" containerName="istio-proxy" Sep 10 23:54:34.402825 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.402799 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0558f6dd-9393-4fbc-938e-78699eecb86f" containerName="istio-proxy" Sep 10 23:54:34.402992 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.402904 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ced2b3c-effb-40be-b02b-9a7aff37a4b0" containerName="istio-proxy" Sep 10 23:54:34.407326 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.407304 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592" Sep 10 23:54:34.410836 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.410809 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ff0028b6\"" Sep 10 23:54:34.410962 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.410841 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ff0028b6-dockercfg-b2q29\"" Sep 10 23:54:34.415856 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.415837 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592"] Sep 10 23:54:34.432724 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.432699 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw"] Sep 10 23:54:34.436144 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.436124 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw" Sep 10 23:54:34.446770 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.446744 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw"] Sep 10 23:54:34.481702 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.481673 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9r4c\" (UniqueName: \"kubernetes.io/projected/2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc-kube-api-access-h9r4c\") pod \"payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw\" (UID: \"2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc\") " pod="openshift-ingress/payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw" Sep 10 23:54:34.481837 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.481729 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqst7\" (UniqueName: \"kubernetes.io/projected/492feec0-17ab-4a6c-b362-368a3853052d-kube-api-access-zqst7\") pod \"payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592\" (UID: \"492feec0-17ab-4a6c-b362-368a3853052d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592" Sep 10 23:54:34.481888 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.481840 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/492feec0-17ab-4a6c-b362-368a3853052d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592\" (UID: \"492feec0-17ab-4a6c-b362-368a3853052d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592" Sep 10 23:54:34.481888 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.481882 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc-plugins-config-volume\") pod \"payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw\" (UID: \"2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc\") " pod="openshift-ingress/payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw" Sep 10 23:54:34.583187 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.583150 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/492feec0-17ab-4a6c-b362-368a3853052d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592\" (UID: \"492feec0-17ab-4a6c-b362-368a3853052d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592" Sep 10 23:54:34.583364 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.583202 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc-plugins-config-volume\") pod \"payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw\" (UID: \"2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc\") " pod="openshift-ingress/payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw" Sep 10 23:54:34.583364 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.583233 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9r4c\" (UniqueName: \"kubernetes.io/projected/2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc-kube-api-access-h9r4c\") pod \"payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw\" (UID: \"2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc\") " pod="openshift-ingress/payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw" Sep 10 23:54:34.583459 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.583349 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqst7\" (UniqueName: \"kubernetes.io/projected/492feec0-17ab-4a6c-b362-368a3853052d-kube-api-access-zqst7\") pod \"payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592\" (UID: \"492feec0-17ab-4a6c-b362-368a3853052d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592" Sep 10 23:54:34.583871 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.583853 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/492feec0-17ab-4a6c-b362-368a3853052d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592\" (UID: \"492feec0-17ab-4a6c-b362-368a3853052d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592" Sep 10 23:54:34.584009 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.583992 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc-plugins-config-volume\") pod \"payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw\" (UID: \"2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc\") " pod="openshift-ingress/payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw" Sep 10 23:54:34.597020 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.596998 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9r4c\" (UniqueName: \"kubernetes.io/projected/2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc-kube-api-access-h9r4c\") pod \"payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw\" (UID: \"2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc\") " pod="openshift-ingress/payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw" Sep 10 23:54:34.601689 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.601671 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqst7\" (UniqueName: \"kubernetes.io/projected/492feec0-17ab-4a6c-b362-368a3853052d-kube-api-access-zqst7\") pod \"payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592\" (UID: \"492feec0-17ab-4a6c-b362-368a3853052d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592" Sep 10 23:54:34.718839 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.718753 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592" Sep 10 23:54:34.746133 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.746103 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw" Sep 10 23:54:34.998409 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:34.998384 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592"] Sep 10 23:54:35.000078 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:54:35.000051 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492feec0_17ab_4a6c_b362_368a3853052d.slice/crio-21229ce5154534e21ff9b66eeda99ba385d05fcc4afb8934b082f42c29befbc2 WatchSource:0}: Error finding container 21229ce5154534e21ff9b66eeda99ba385d05fcc4afb8934b082f42c29befbc2: Status 404 returned error can't find the container with id 21229ce5154534e21ff9b66eeda99ba385d05fcc4afb8934b082f42c29befbc2 Sep 10 23:54:35.026227 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:35.026203 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw"] Sep 10 23:54:35.029104 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:54:35.029077 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea8d2ee_5861_4da6_8a12_f6cff2bbe0bc.slice/crio-f03f81cbede6a606a0e0aaa565ee8be2eb923bc7b9deccfd6a2d8df75fe13641 WatchSource:0}: Error finding container f03f81cbede6a606a0e0aaa565ee8be2eb923bc7b9deccfd6a2d8df75fe13641: Status 404 returned error can't find the container with id f03f81cbede6a606a0e0aaa565ee8be2eb923bc7b9deccfd6a2d8df75fe13641 Sep 10 23:54:35.766140 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:35.766107 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592" event={"ID":"492feec0-17ab-4a6c-b362-368a3853052d","Type":"ContainerStarted","Data":"46effe49ae54a82e7d6a61f971c4c99168be4aa1f19635f78741599ba46c4c29"} Sep 10 23:54:35.766694 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:35.766158 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592" event={"ID":"492feec0-17ab-4a6c-b362-368a3853052d","Type":"ContainerStarted","Data":"21229ce5154534e21ff9b66eeda99ba385d05fcc4afb8934b082f42c29befbc2"} Sep 10 23:54:35.766694 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:35.766279 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592" Sep 10 23:54:35.767724 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:35.767698 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw" event={"ID":"2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc","Type":"ContainerStarted","Data":"3b2051aac4d981ce2e15918e0e82a65c30d84c477ce70b2b1865097079b3ff8f"} Sep 10 23:54:35.767829 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:35.767737 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw" event={"ID":"2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc","Type":"ContainerStarted","Data":"f03f81cbede6a606a0e0aaa565ee8be2eb923bc7b9deccfd6a2d8df75fe13641"} Sep 10 23:54:35.767829 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:35.767758 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw" Sep 10 23:54:35.781757 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:35.781705 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592" podStartSLOduration=1.781686359 podStartE2EDuration="1.781686359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:54:35.78133832 +0000 UTC m=+1091.248812668" watchObservedRunningTime="2026-09-10 23:54:35.781686359 +0000 UTC m=+1091.249160685" Sep 10 23:54:35.798268 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:35.798219 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw" podStartSLOduration=1.798203157 podStartE2EDuration="1.798203157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:54:35.797439062 +0000 UTC m=+1091.264913388" watchObservedRunningTime="2026-09-10 23:54:35.798203157 +0000 UTC m=+1091.265677481" Sep 10 23:54:46.778106 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:46.778073 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw" Sep 10 23:54:46.778585 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:54:46.778234 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592" Sep 10 23:55:11.388159 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.388119 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm"] Sep 10 23:55:11.388728 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.388340 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm" podUID="ff709b92-c05f-41b6-8a5c-e0f59549ecd7" containerName="payload-processing" containerID="cri-o://d92444167ebf41f2f3a2dd2b464b7359d0ee281cf26fc38ae91e29ee54682590" gracePeriod=30 Sep 10 23:55:11.395008 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.394980 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25"] Sep 10 23:55:11.395256 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.395227 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25" podUID="f42dfd8b-5bb6-4121-994a-0a5ff684470a" containerName="payload-pre-processing" containerID="cri-o://939f8b82d75a1b557b97fa4217e663ad38521bf50350fa3f4203cd1d8838d22a" gracePeriod=30 Sep 10 23:55:11.648728 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.648673 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm" Sep 10 23:55:11.673965 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.673927 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25" Sep 10 23:55:11.708804 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.708777 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff709b92-c05f-41b6-8a5c-e0f59549ecd7-plugins-config-volume\") pod \"ff709b92-c05f-41b6-8a5c-e0f59549ecd7\" (UID: \"ff709b92-c05f-41b6-8a5c-e0f59549ecd7\") " Sep 10 23:55:11.708970 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.708861 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f42dfd8b-5bb6-4121-994a-0a5ff684470a-plugins-config-volume\") pod \"f42dfd8b-5bb6-4121-994a-0a5ff684470a\" (UID: \"f42dfd8b-5bb6-4121-994a-0a5ff684470a\") " Sep 10 23:55:11.708970 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.708880 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4smvj\" (UniqueName: \"kubernetes.io/projected/ff709b92-c05f-41b6-8a5c-e0f59549ecd7-kube-api-access-4smvj\") pod \"ff709b92-c05f-41b6-8a5c-e0f59549ecd7\" (UID: \"ff709b92-c05f-41b6-8a5c-e0f59549ecd7\") " Sep 10 23:55:11.708970 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.708901 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4bx5\" (UniqueName: \"kubernetes.io/projected/f42dfd8b-5bb6-4121-994a-0a5ff684470a-kube-api-access-t4bx5\") pod \"f42dfd8b-5bb6-4121-994a-0a5ff684470a\" (UID: \"f42dfd8b-5bb6-4121-994a-0a5ff684470a\") " Sep 10 23:55:11.709251 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.709221 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff709b92-c05f-41b6-8a5c-e0f59549ecd7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ff709b92-c05f-41b6-8a5c-e0f59549ecd7" (UID: "ff709b92-c05f-41b6-8a5c-e0f59549ecd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:55:11.709322 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.709283 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f42dfd8b-5bb6-4121-994a-0a5ff684470a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f42dfd8b-5bb6-4121-994a-0a5ff684470a" (UID: "f42dfd8b-5bb6-4121-994a-0a5ff684470a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:55:11.711130 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.711102 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff709b92-c05f-41b6-8a5c-e0f59549ecd7-kube-api-access-4smvj" (OuterVolumeSpecName: "kube-api-access-4smvj") pod "ff709b92-c05f-41b6-8a5c-e0f59549ecd7" (UID: "ff709b92-c05f-41b6-8a5c-e0f59549ecd7"). InnerVolumeSpecName "kube-api-access-4smvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:55:11.711217 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.711137 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f42dfd8b-5bb6-4121-994a-0a5ff684470a-kube-api-access-t4bx5" (OuterVolumeSpecName: "kube-api-access-t4bx5") pod "f42dfd8b-5bb6-4121-994a-0a5ff684470a" (UID: "f42dfd8b-5bb6-4121-994a-0a5ff684470a"). InnerVolumeSpecName "kube-api-access-t4bx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:55:11.810262 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.810221 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff709b92-c05f-41b6-8a5c-e0f59549ecd7-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:55:11.810262 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.810254 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f42dfd8b-5bb6-4121-994a-0a5ff684470a-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:55:11.810262 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.810263 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4smvj\" (UniqueName: \"kubernetes.io/projected/ff709b92-c05f-41b6-8a5c-e0f59549ecd7-kube-api-access-4smvj\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:55:11.810262 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.810271 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4bx5\" (UniqueName: \"kubernetes.io/projected/f42dfd8b-5bb6-4121-994a-0a5ff684470a-kube-api-access-t4bx5\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:55:11.940993 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.940878 2578 generic.go:356] "Generic (PLEG): container finished" podID="ff709b92-c05f-41b6-8a5c-e0f59549ecd7" containerID="d92444167ebf41f2f3a2dd2b464b7359d0ee281cf26fc38ae91e29ee54682590" exitCode=0 Sep 10 23:55:11.940993 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.940915 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm" Sep 10 23:55:11.940993 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.940959 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm" event={"ID":"ff709b92-c05f-41b6-8a5c-e0f59549ecd7","Type":"ContainerDied","Data":"d92444167ebf41f2f3a2dd2b464b7359d0ee281cf26fc38ae91e29ee54682590"} Sep 10 23:55:11.941252 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.941015 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm" event={"ID":"ff709b92-c05f-41b6-8a5c-e0f59549ecd7","Type":"ContainerDied","Data":"8d706c126db39e3ce1e439c4089b4294b1538b0f204a0aaa0cc3eb98184cc0d6"} Sep 10 23:55:11.941252 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.941038 2578 scope.go:117] "RemoveContainer" containerID="d92444167ebf41f2f3a2dd2b464b7359d0ee281cf26fc38ae91e29ee54682590" Sep 10 23:55:11.942930 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.942896 2578 generic.go:356] "Generic (PLEG): container finished" podID="f42dfd8b-5bb6-4121-994a-0a5ff684470a" containerID="939f8b82d75a1b557b97fa4217e663ad38521bf50350fa3f4203cd1d8838d22a" exitCode=0 Sep 10 23:55:11.943117 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.943035 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25" event={"ID":"f42dfd8b-5bb6-4121-994a-0a5ff684470a","Type":"ContainerDied","Data":"939f8b82d75a1b557b97fa4217e663ad38521bf50350fa3f4203cd1d8838d22a"} Sep 10 23:55:11.943235 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.943133 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25" event={"ID":"f42dfd8b-5bb6-4121-994a-0a5ff684470a","Type":"ContainerDied","Data":"276934f1dc69cd08af63257962e6fde8ba2951807a7719a4a561c14c0a2d638a"} Sep 10 23:55:11.943235 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.943226 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25" Sep 10 23:55:11.954466 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.954449 2578 scope.go:117] "RemoveContainer" containerID="d92444167ebf41f2f3a2dd2b464b7359d0ee281cf26fc38ae91e29ee54682590" Sep 10 23:55:11.954758 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:55:11.954741 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d92444167ebf41f2f3a2dd2b464b7359d0ee281cf26fc38ae91e29ee54682590\": container with ID starting with d92444167ebf41f2f3a2dd2b464b7359d0ee281cf26fc38ae91e29ee54682590 not found: ID does not exist" containerID="d92444167ebf41f2f3a2dd2b464b7359d0ee281cf26fc38ae91e29ee54682590" Sep 10 23:55:11.954822 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.954765 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d92444167ebf41f2f3a2dd2b464b7359d0ee281cf26fc38ae91e29ee54682590"} err="failed to get container status \"d92444167ebf41f2f3a2dd2b464b7359d0ee281cf26fc38ae91e29ee54682590\": rpc error: code = NotFound desc = could not find container \"d92444167ebf41f2f3a2dd2b464b7359d0ee281cf26fc38ae91e29ee54682590\": container with ID starting with d92444167ebf41f2f3a2dd2b464b7359d0ee281cf26fc38ae91e29ee54682590 not found: ID does not exist" Sep 10 23:55:11.954822 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.954780 2578 scope.go:117] "RemoveContainer" containerID="939f8b82d75a1b557b97fa4217e663ad38521bf50350fa3f4203cd1d8838d22a" Sep 10 23:55:11.962663 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.962643 2578 scope.go:117] "RemoveContainer" containerID="939f8b82d75a1b557b97fa4217e663ad38521bf50350fa3f4203cd1d8838d22a" Sep 10 23:55:11.962914 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:55:11.962897 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"939f8b82d75a1b557b97fa4217e663ad38521bf50350fa3f4203cd1d8838d22a\": container with ID starting with 939f8b82d75a1b557b97fa4217e663ad38521bf50350fa3f4203cd1d8838d22a not found: ID does not exist" containerID="939f8b82d75a1b557b97fa4217e663ad38521bf50350fa3f4203cd1d8838d22a" Sep 10 23:55:11.962981 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.962920 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"939f8b82d75a1b557b97fa4217e663ad38521bf50350fa3f4203cd1d8838d22a"} err="failed to get container status \"939f8b82d75a1b557b97fa4217e663ad38521bf50350fa3f4203cd1d8838d22a\": rpc error: code = NotFound desc = could not find container \"939f8b82d75a1b557b97fa4217e663ad38521bf50350fa3f4203cd1d8838d22a\": container with ID starting with 939f8b82d75a1b557b97fa4217e663ad38521bf50350fa3f4203cd1d8838d22a not found: ID does not exist" Sep 10 23:55:11.968413 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.968389 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm"] Sep 10 23:55:11.972444 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.972426 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ce1dd5-88b68c469-95gdm"] Sep 10 23:55:11.987136 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.987113 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25"] Sep 10 23:55:11.989776 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:11.989756 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ce1dd5-5b9f8685db-p7d25"] Sep 10 23:55:13.043964 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:13.043918 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f42dfd8b-5bb6-4121-994a-0a5ff684470a" path="/var/lib/kubelet/pods/f42dfd8b-5bb6-4121-994a-0a5ff684470a/volumes" Sep 10 23:55:13.044319 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:13.044283 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff709b92-c05f-41b6-8a5c-e0f59549ecd7" path="/var/lib/kubelet/pods/ff709b92-c05f-41b6-8a5c-e0f59549ecd7/volumes" Sep 10 23:55:15.902081 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:15.902032 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw"] Sep 10 23:55:15.902541 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:15.902285 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw" podUID="2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc" containerName="payload-processing" containerID="cri-o://3b2051aac4d981ce2e15918e0e82a65c30d84c477ce70b2b1865097079b3ff8f" gracePeriod=30 Sep 10 23:55:15.911304 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:15.911026 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592"] Sep 10 23:55:15.911304 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:15.911244 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592" podUID="492feec0-17ab-4a6c-b362-368a3853052d" containerName="payload-pre-processing" containerID="cri-o://46effe49ae54a82e7d6a61f971c4c99168be4aa1f19635f78741599ba46c4c29" gracePeriod=30 Sep 10 23:55:16.178560 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.178538 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw" Sep 10 23:55:16.180685 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.180668 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592" Sep 10 23:55:16.245093 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.245060 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9r4c\" (UniqueName: \"kubernetes.io/projected/2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc-kube-api-access-h9r4c\") pod \"2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc\" (UID: \"2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc\") " Sep 10 23:55:16.245273 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.245160 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqst7\" (UniqueName: \"kubernetes.io/projected/492feec0-17ab-4a6c-b362-368a3853052d-kube-api-access-zqst7\") pod \"492feec0-17ab-4a6c-b362-368a3853052d\" (UID: \"492feec0-17ab-4a6c-b362-368a3853052d\") " Sep 10 23:55:16.245273 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.245208 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/492feec0-17ab-4a6c-b362-368a3853052d-plugins-config-volume\") pod \"492feec0-17ab-4a6c-b362-368a3853052d\" (UID: \"492feec0-17ab-4a6c-b362-368a3853052d\") " Sep 10 23:55:16.245273 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.245227 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc-plugins-config-volume\") pod \"2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc\" (UID: \"2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc\") " Sep 10 23:55:16.245557 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.245524 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/492feec0-17ab-4a6c-b362-368a3853052d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "492feec0-17ab-4a6c-b362-368a3853052d" (UID: "492feec0-17ab-4a6c-b362-368a3853052d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:55:16.245631 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.245611 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc" (UID: "2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:55:16.247160 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.247136 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc-kube-api-access-h9r4c" (OuterVolumeSpecName: "kube-api-access-h9r4c") pod "2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc" (UID: "2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc"). InnerVolumeSpecName "kube-api-access-h9r4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:55:16.247260 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.247221 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492feec0-17ab-4a6c-b362-368a3853052d-kube-api-access-zqst7" (OuterVolumeSpecName: "kube-api-access-zqst7") pod "492feec0-17ab-4a6c-b362-368a3853052d" (UID: "492feec0-17ab-4a6c-b362-368a3853052d"). InnerVolumeSpecName "kube-api-access-zqst7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:55:16.346812 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.346781 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9r4c\" (UniqueName: \"kubernetes.io/projected/2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc-kube-api-access-h9r4c\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:55:16.346812 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.346808 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqst7\" (UniqueName: \"kubernetes.io/projected/492feec0-17ab-4a6c-b362-368a3853052d-kube-api-access-zqst7\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:55:16.346994 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.346822 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/492feec0-17ab-4a6c-b362-368a3853052d-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:55:16.346994 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.346831 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:55:16.972174 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.972134 2578 generic.go:356] "Generic (PLEG): container finished" podID="2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc" containerID="3b2051aac4d981ce2e15918e0e82a65c30d84c477ce70b2b1865097079b3ff8f" exitCode=0 Sep 10 23:55:16.972174 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.972177 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw" Sep 10 23:55:16.972695 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.972225 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw" event={"ID":"2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc","Type":"ContainerDied","Data":"3b2051aac4d981ce2e15918e0e82a65c30d84c477ce70b2b1865097079b3ff8f"} Sep 10 23:55:16.972695 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.972283 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw" event={"ID":"2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc","Type":"ContainerDied","Data":"f03f81cbede6a606a0e0aaa565ee8be2eb923bc7b9deccfd6a2d8df75fe13641"} Sep 10 23:55:16.972695 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.972308 2578 scope.go:117] "RemoveContainer" containerID="3b2051aac4d981ce2e15918e0e82a65c30d84c477ce70b2b1865097079b3ff8f" Sep 10 23:55:16.976429 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.975536 2578 generic.go:356] "Generic (PLEG): container finished" podID="492feec0-17ab-4a6c-b362-368a3853052d" containerID="46effe49ae54a82e7d6a61f971c4c99168be4aa1f19635f78741599ba46c4c29" exitCode=0 Sep 10 23:55:16.976429 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.975589 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592" event={"ID":"492feec0-17ab-4a6c-b362-368a3853052d","Type":"ContainerDied","Data":"46effe49ae54a82e7d6a61f971c4c99168be4aa1f19635f78741599ba46c4c29"} Sep 10 23:55:16.976429 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.975673 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592" event={"ID":"492feec0-17ab-4a6c-b362-368a3853052d","Type":"ContainerDied","Data":"21229ce5154534e21ff9b66eeda99ba385d05fcc4afb8934b082f42c29befbc2"} Sep 10 23:55:16.976429 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.975772 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592" Sep 10 23:55:16.989299 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.989281 2578 scope.go:117] "RemoveContainer" containerID="3b2051aac4d981ce2e15918e0e82a65c30d84c477ce70b2b1865097079b3ff8f" Sep 10 23:55:16.989559 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:55:16.989536 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b2051aac4d981ce2e15918e0e82a65c30d84c477ce70b2b1865097079b3ff8f\": container with ID starting with 3b2051aac4d981ce2e15918e0e82a65c30d84c477ce70b2b1865097079b3ff8f not found: ID does not exist" containerID="3b2051aac4d981ce2e15918e0e82a65c30d84c477ce70b2b1865097079b3ff8f" Sep 10 23:55:16.989701 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.989565 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2051aac4d981ce2e15918e0e82a65c30d84c477ce70b2b1865097079b3ff8f"} err="failed to get container status \"3b2051aac4d981ce2e15918e0e82a65c30d84c477ce70b2b1865097079b3ff8f\": rpc error: code = NotFound desc = could not find container \"3b2051aac4d981ce2e15918e0e82a65c30d84c477ce70b2b1865097079b3ff8f\": container with ID starting with 3b2051aac4d981ce2e15918e0e82a65c30d84c477ce70b2b1865097079b3ff8f not found: ID does not exist" Sep 10 23:55:16.989701 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.989581 2578 scope.go:117] "RemoveContainer" containerID="46effe49ae54a82e7d6a61f971c4c99168be4aa1f19635f78741599ba46c4c29" Sep 10 23:55:16.997384 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.997368 2578 scope.go:117] "RemoveContainer" containerID="46effe49ae54a82e7d6a61f971c4c99168be4aa1f19635f78741599ba46c4c29" Sep 10 23:55:16.997450 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.997370 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw"] Sep 10 23:55:16.997650 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:55:16.997634 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46effe49ae54a82e7d6a61f971c4c99168be4aa1f19635f78741599ba46c4c29\": container with ID starting with 46effe49ae54a82e7d6a61f971c4c99168be4aa1f19635f78741599ba46c4c29 not found: ID does not exist" containerID="46effe49ae54a82e7d6a61f971c4c99168be4aa1f19635f78741599ba46c4c29" Sep 10 23:55:16.997693 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:16.997655 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46effe49ae54a82e7d6a61f971c4c99168be4aa1f19635f78741599ba46c4c29"} err="failed to get container status \"46effe49ae54a82e7d6a61f971c4c99168be4aa1f19635f78741599ba46c4c29\": rpc error: code = NotFound desc = could not find container \"46effe49ae54a82e7d6a61f971c4c99168be4aa1f19635f78741599ba46c4c29\": container with ID starting with 46effe49ae54a82e7d6a61f971c4c99168be4aa1f19635f78741599ba46c4c29 not found: ID does not exist" Sep 10 23:55:17.001075 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:17.001055 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ff0028b6-6d8766465-f44gw"] Sep 10 23:55:17.014830 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:17.014806 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592"] Sep 10 23:55:17.017828 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:17.017808 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ff0028b6-77f4fd8db-6z592"] Sep 10 23:55:17.043967 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:17.043927 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc" path="/var/lib/kubelet/pods/2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc/volumes" Sep 10 23:55:17.044371 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:17.044353 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="492feec0-17ab-4a6c-b362-368a3853052d" path="/var/lib/kubelet/pods/492feec0-17ab-4a6c-b362-368a3853052d/volumes" Sep 10 23:55:18.087846 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:18.087815 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d"] Sep 10 23:55:18.088306 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:18.088078 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" podUID="2a673735-8cb8-4342-81a1-e7ec4357fbf7" containerName="istio-proxy" containerID="cri-o://fd894e883ad79133f6d2e7a4ed6f1cf1cfdcceeb4274c7cf6a6b27c090ec1e74" gracePeriod=30 Sep 10 23:55:23.344343 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.344317 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:55:23.406218 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.406188 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istio-token\") pod \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " Sep 10 23:55:23.406368 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.406283 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-workload-socket\") pod \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " Sep 10 23:55:23.406368 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.406336 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istio-data\") pod \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " Sep 10 23:55:23.406459 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.406377 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istiod-ca-cert\") pod \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " Sep 10 23:55:23.406459 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.406401 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-credential-socket\") pod \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " Sep 10 23:55:23.406459 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.406451 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-workload-certs\") pod \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " Sep 10 23:55:23.406596 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.406483 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhr2s\" (UniqueName: \"kubernetes.io/projected/2a673735-8cb8-4342-81a1-e7ec4357fbf7-kube-api-access-xhr2s\") pod \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " Sep 10 23:55:23.406596 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.406527 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istio-envoy\") pod \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " Sep 10 23:55:23.406596 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.406552 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istio-podinfo\") pod \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\" (UID: \"2a673735-8cb8-4342-81a1-e7ec4357fbf7\") " Sep 10 23:55:23.406742 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.406621 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2a673735-8cb8-4342-81a1-e7ec4357fbf7" (UID: "2a673735-8cb8-4342-81a1-e7ec4357fbf7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:55:23.406742 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.406706 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2a673735-8cb8-4342-81a1-e7ec4357fbf7" (UID: "2a673735-8cb8-4342-81a1-e7ec4357fbf7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:55:23.406845 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.406634 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istio-data" (OuterVolumeSpecName: "istio-data") pod "2a673735-8cb8-4342-81a1-e7ec4357fbf7" (UID: "2a673735-8cb8-4342-81a1-e7ec4357fbf7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:55:23.406905 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.406864 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2a673735-8cb8-4342-81a1-e7ec4357fbf7" (UID: "2a673735-8cb8-4342-81a1-e7ec4357fbf7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:55:23.406995 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.406924 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-workload-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:55:23.406995 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.406970 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istio-data\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:55:23.406995 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.406984 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-credential-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:55:23.407163 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.407132 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2a673735-8cb8-4342-81a1-e7ec4357fbf7" (UID: "2a673735-8cb8-4342-81a1-e7ec4357fbf7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:55:23.408449 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.408422 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istio-token" (OuterVolumeSpecName: "istio-token") pod "2a673735-8cb8-4342-81a1-e7ec4357fbf7" (UID: "2a673735-8cb8-4342-81a1-e7ec4357fbf7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:55:23.408841 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.408827 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2a673735-8cb8-4342-81a1-e7ec4357fbf7" (UID: "2a673735-8cb8-4342-81a1-e7ec4357fbf7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 23:55:23.409097 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.409077 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2a673735-8cb8-4342-81a1-e7ec4357fbf7" (UID: "2a673735-8cb8-4342-81a1-e7ec4357fbf7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:55:23.409190 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.409093 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a673735-8cb8-4342-81a1-e7ec4357fbf7-kube-api-access-xhr2s" (OuterVolumeSpecName: "kube-api-access-xhr2s") pod "2a673735-8cb8-4342-81a1-e7ec4357fbf7" (UID: "2a673735-8cb8-4342-81a1-e7ec4357fbf7"). InnerVolumeSpecName "kube-api-access-xhr2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:55:23.508271 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.508242 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istiod-ca-cert\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:55:23.508271 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.508266 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-workload-certs\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:55:23.508271 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.508275 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhr2s\" (UniqueName: \"kubernetes.io/projected/2a673735-8cb8-4342-81a1-e7ec4357fbf7-kube-api-access-xhr2s\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:55:23.508477 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.508285 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istio-envoy\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:55:23.508477 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.508293 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istio-podinfo\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:55:23.508477 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:23.508300 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a673735-8cb8-4342-81a1-e7ec4357fbf7-istio-token\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:55:24.011663 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:24.011625 2578 generic.go:356] "Generic (PLEG): container finished" podID="2a673735-8cb8-4342-81a1-e7ec4357fbf7" containerID="fd894e883ad79133f6d2e7a4ed6f1cf1cfdcceeb4274c7cf6a6b27c090ec1e74" exitCode=0 Sep 10 23:55:24.011810 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:24.011674 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" Sep 10 23:55:24.011810 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:24.011686 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" event={"ID":"2a673735-8cb8-4342-81a1-e7ec4357fbf7","Type":"ContainerDied","Data":"fd894e883ad79133f6d2e7a4ed6f1cf1cfdcceeb4274c7cf6a6b27c090ec1e74"} Sep 10 23:55:24.011810 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:24.011737 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d" event={"ID":"2a673735-8cb8-4342-81a1-e7ec4357fbf7","Type":"ContainerDied","Data":"54a8114335ef5ca6584edc0718ef61cec98d74956966713dc598cfa203f2c45a"} Sep 10 23:55:24.011810 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:24.011766 2578 scope.go:117] "RemoveContainer" containerID="fd894e883ad79133f6d2e7a4ed6f1cf1cfdcceeb4274c7cf6a6b27c090ec1e74" Sep 10 23:55:24.020659 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:24.020640 2578 scope.go:117] "RemoveContainer" containerID="bacc0c57698050fc456b6ccc247511aa2bcf43c84c780d822a9a68302d7f01fa" Sep 10 23:55:24.028052 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:24.028034 2578 scope.go:117] "RemoveContainer" containerID="fd894e883ad79133f6d2e7a4ed6f1cf1cfdcceeb4274c7cf6a6b27c090ec1e74" Sep 10 23:55:24.028640 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:55:24.028619 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd894e883ad79133f6d2e7a4ed6f1cf1cfdcceeb4274c7cf6a6b27c090ec1e74\": container with ID starting with fd894e883ad79133f6d2e7a4ed6f1cf1cfdcceeb4274c7cf6a6b27c090ec1e74 not found: ID does not exist" containerID="fd894e883ad79133f6d2e7a4ed6f1cf1cfdcceeb4274c7cf6a6b27c090ec1e74" Sep 10 23:55:24.028726 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:24.028647 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd894e883ad79133f6d2e7a4ed6f1cf1cfdcceeb4274c7cf6a6b27c090ec1e74"} err="failed to get container status \"fd894e883ad79133f6d2e7a4ed6f1cf1cfdcceeb4274c7cf6a6b27c090ec1e74\": rpc error: code = NotFound desc = could not find container \"fd894e883ad79133f6d2e7a4ed6f1cf1cfdcceeb4274c7cf6a6b27c090ec1e74\": container with ID starting with fd894e883ad79133f6d2e7a4ed6f1cf1cfdcceeb4274c7cf6a6b27c090ec1e74 not found: ID does not exist" Sep 10 23:55:24.028726 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:24.028666 2578 scope.go:117] "RemoveContainer" containerID="bacc0c57698050fc456b6ccc247511aa2bcf43c84c780d822a9a68302d7f01fa" Sep 10 23:55:24.028909 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:55:24.028894 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bacc0c57698050fc456b6ccc247511aa2bcf43c84c780d822a9a68302d7f01fa\": container with ID starting with bacc0c57698050fc456b6ccc247511aa2bcf43c84c780d822a9a68302d7f01fa not found: ID does not exist" containerID="bacc0c57698050fc456b6ccc247511aa2bcf43c84c780d822a9a68302d7f01fa" Sep 10 23:55:24.028974 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:24.028912 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bacc0c57698050fc456b6ccc247511aa2bcf43c84c780d822a9a68302d7f01fa"} err="failed to get container status \"bacc0c57698050fc456b6ccc247511aa2bcf43c84c780d822a9a68302d7f01fa\": rpc error: code = NotFound desc = could not find container \"bacc0c57698050fc456b6ccc247511aa2bcf43c84c780d822a9a68302d7f01fa\": container with ID starting with bacc0c57698050fc456b6ccc247511aa2bcf43c84c780d822a9a68302d7f01fa not found: ID does not exist" Sep 10 23:55:24.033571 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:24.033545 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d"] Sep 10 23:55:24.037829 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:24.037804 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ce1dd5-openshift-default-5dffccfbf4-47j4d"] Sep 10 23:55:25.043266 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:25.043233 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a673735-8cb8-4342-81a1-e7ec4357fbf7" path="/var/lib/kubelet/pods/2a673735-8cb8-4342-81a1-e7ec4357fbf7/volumes" Sep 10 23:55:29.056799 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:29.056764 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg"] Sep 10 23:55:29.057348 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:29.057297 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg" podUID="ea312b45-cf18-4c8d-a7e6-10a5ef475ee4" containerName="payload-processing" containerID="cri-o://061628c7fccfc66253f14ac07924e9f9ac0068e364e644da53c5e429531afe63" gracePeriod=30 Sep 10 23:55:29.066098 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:29.066074 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj"] Sep 10 23:55:29.066335 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:29.066301 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj" podUID="ecbe850b-7c2f-4556-b6b5-f1aa2ec62703" containerName="payload-pre-processing" containerID="cri-o://c4ff20d27dd49e6ffe501def82008d2d03e8852a15cba0346102e17345016914" gracePeriod=30 Sep 10 23:55:29.330406 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:29.330379 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg" Sep 10 23:55:29.332984 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:29.332967 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj" Sep 10 23:55:29.461868 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:29.461834 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97npn\" (UniqueName: \"kubernetes.io/projected/ecbe850b-7c2f-4556-b6b5-f1aa2ec62703-kube-api-access-97npn\") pod \"ecbe850b-7c2f-4556-b6b5-f1aa2ec62703\" (UID: \"ecbe850b-7c2f-4556-b6b5-f1aa2ec62703\") " Sep 10 23:55:29.462072 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:29.461899 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4tdm\" (UniqueName: \"kubernetes.io/projected/ea312b45-cf18-4c8d-a7e6-10a5ef475ee4-kube-api-access-x4tdm\") pod \"ea312b45-cf18-4c8d-a7e6-10a5ef475ee4\" (UID: \"ea312b45-cf18-4c8d-a7e6-10a5ef475ee4\") " Sep 10 23:55:29.462072 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:29.462002 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea312b45-cf18-4c8d-a7e6-10a5ef475ee4-plugins-config-volume\") pod \"ea312b45-cf18-4c8d-a7e6-10a5ef475ee4\" (UID: \"ea312b45-cf18-4c8d-a7e6-10a5ef475ee4\") " Sep 10 23:55:29.462072 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:29.462025 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbe850b-7c2f-4556-b6b5-f1aa2ec62703-plugins-config-volume\") pod \"ecbe850b-7c2f-4556-b6b5-f1aa2ec62703\" (UID: \"ecbe850b-7c2f-4556-b6b5-f1aa2ec62703\") " Sep 10 23:55:29.462383 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:29.462360 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea312b45-cf18-4c8d-a7e6-10a5ef475ee4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ea312b45-cf18-4c8d-a7e6-10a5ef475ee4" (UID: "ea312b45-cf18-4c8d-a7e6-10a5ef475ee4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:55:29.462466 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:29.462423 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecbe850b-7c2f-4556-b6b5-f1aa2ec62703-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ecbe850b-7c2f-4556-b6b5-f1aa2ec62703" (UID: "ecbe850b-7c2f-4556-b6b5-f1aa2ec62703"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:55:29.463972 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:29.463926 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecbe850b-7c2f-4556-b6b5-f1aa2ec62703-kube-api-access-97npn" (OuterVolumeSpecName: "kube-api-access-97npn") pod "ecbe850b-7c2f-4556-b6b5-f1aa2ec62703" (UID: "ecbe850b-7c2f-4556-b6b5-f1aa2ec62703"). InnerVolumeSpecName "kube-api-access-97npn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:55:29.463972 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:29.463962 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea312b45-cf18-4c8d-a7e6-10a5ef475ee4-kube-api-access-x4tdm" (OuterVolumeSpecName: "kube-api-access-x4tdm") pod "ea312b45-cf18-4c8d-a7e6-10a5ef475ee4" (UID: "ea312b45-cf18-4c8d-a7e6-10a5ef475ee4"). InnerVolumeSpecName "kube-api-access-x4tdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:55:29.562662 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:29.562585 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97npn\" (UniqueName: \"kubernetes.io/projected/ecbe850b-7c2f-4556-b6b5-f1aa2ec62703-kube-api-access-97npn\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:55:29.562662 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:29.562613 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4tdm\" (UniqueName: \"kubernetes.io/projected/ea312b45-cf18-4c8d-a7e6-10a5ef475ee4-kube-api-access-x4tdm\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:55:29.562662 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:29.562623 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea312b45-cf18-4c8d-a7e6-10a5ef475ee4-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:55:29.562662 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:29.562632 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbe850b-7c2f-4556-b6b5-f1aa2ec62703-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:55:30.040921 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:30.040880 2578 generic.go:356] "Generic (PLEG): container finished" podID="ecbe850b-7c2f-4556-b6b5-f1aa2ec62703" containerID="c4ff20d27dd49e6ffe501def82008d2d03e8852a15cba0346102e17345016914" exitCode=0 Sep 10 23:55:30.040921 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:30.040918 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj" Sep 10 23:55:30.041172 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:30.040970 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj" event={"ID":"ecbe850b-7c2f-4556-b6b5-f1aa2ec62703","Type":"ContainerDied","Data":"c4ff20d27dd49e6ffe501def82008d2d03e8852a15cba0346102e17345016914"} Sep 10 23:55:30.041172 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:30.041014 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj" event={"ID":"ecbe850b-7c2f-4556-b6b5-f1aa2ec62703","Type":"ContainerDied","Data":"6dc87fb689e49dc9cdd93c4d86b971f7489426e04f7dc2b4f3f04be2461278b2"} Sep 10 23:55:30.041172 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:30.041037 2578 scope.go:117] "RemoveContainer" containerID="c4ff20d27dd49e6ffe501def82008d2d03e8852a15cba0346102e17345016914" Sep 10 23:55:30.042257 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:30.042226 2578 generic.go:356] "Generic (PLEG): container finished" podID="ea312b45-cf18-4c8d-a7e6-10a5ef475ee4" containerID="061628c7fccfc66253f14ac07924e9f9ac0068e364e644da53c5e429531afe63" exitCode=0 Sep 10 23:55:30.042337 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:30.042258 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg" Sep 10 23:55:30.042337 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:30.042310 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg" event={"ID":"ea312b45-cf18-4c8d-a7e6-10a5ef475ee4","Type":"ContainerDied","Data":"061628c7fccfc66253f14ac07924e9f9ac0068e364e644da53c5e429531afe63"} Sep 10 23:55:30.042436 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:30.042346 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg" event={"ID":"ea312b45-cf18-4c8d-a7e6-10a5ef475ee4","Type":"ContainerDied","Data":"93ece356570d1094ce8022722530989c383196628be5c1deb1ce16161401ff0f"} Sep 10 23:55:30.050389 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:30.050380 2578 scope.go:117] "RemoveContainer" containerID="c4ff20d27dd49e6ffe501def82008d2d03e8852a15cba0346102e17345016914" Sep 10 23:55:30.050703 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:55:30.050660 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4ff20d27dd49e6ffe501def82008d2d03e8852a15cba0346102e17345016914\": container with ID starting with c4ff20d27dd49e6ffe501def82008d2d03e8852a15cba0346102e17345016914 not found: ID does not exist" containerID="c4ff20d27dd49e6ffe501def82008d2d03e8852a15cba0346102e17345016914" Sep 10 23:55:30.050772 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:30.050694 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4ff20d27dd49e6ffe501def82008d2d03e8852a15cba0346102e17345016914"} err="failed to get container status \"c4ff20d27dd49e6ffe501def82008d2d03e8852a15cba0346102e17345016914\": rpc error: code = NotFound desc = could not find container \"c4ff20d27dd49e6ffe501def82008d2d03e8852a15cba0346102e17345016914\": container with ID starting with c4ff20d27dd49e6ffe501def82008d2d03e8852a15cba0346102e17345016914 not found: ID does not exist" Sep 10 23:55:30.050772 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:30.050716 2578 scope.go:117] "RemoveContainer" containerID="061628c7fccfc66253f14ac07924e9f9ac0068e364e644da53c5e429531afe63" Sep 10 23:55:30.058600 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:30.058426 2578 scope.go:117] "RemoveContainer" containerID="061628c7fccfc66253f14ac07924e9f9ac0068e364e644da53c5e429531afe63" Sep 10 23:55:30.058847 ip-10-0-139-142 kubenswrapper[2578]: E0910 23:55:30.058677 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"061628c7fccfc66253f14ac07924e9f9ac0068e364e644da53c5e429531afe63\": container with ID starting with 061628c7fccfc66253f14ac07924e9f9ac0068e364e644da53c5e429531afe63 not found: ID does not exist" containerID="061628c7fccfc66253f14ac07924e9f9ac0068e364e644da53c5e429531afe63" Sep 10 23:55:30.058847 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:30.058710 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"061628c7fccfc66253f14ac07924e9f9ac0068e364e644da53c5e429531afe63"} err="failed to get container status \"061628c7fccfc66253f14ac07924e9f9ac0068e364e644da53c5e429531afe63\": rpc error: code = NotFound desc = could not find container \"061628c7fccfc66253f14ac07924e9f9ac0068e364e644da53c5e429531afe63\": container with ID starting with 061628c7fccfc66253f14ac07924e9f9ac0068e364e644da53c5e429531afe63 not found: ID does not exist" Sep 10 23:55:30.064926 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:30.064901 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj"] Sep 10 23:55:30.068255 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:30.068236 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-13374d-6bd4fc48f4-hrxmj"] Sep 10 23:55:30.077731 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:30.077710 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg"] Sep 10 23:55:30.079740 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:30.079721 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-13374d-7fc84c75cf-d4slg"] Sep 10 23:55:31.046035 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:31.046001 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea312b45-cf18-4c8d-a7e6-10a5ef475ee4" path="/var/lib/kubelet/pods/ea312b45-cf18-4c8d-a7e6-10a5ef475ee4/volumes" Sep 10 23:55:31.046468 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:31.046449 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecbe850b-7c2f-4556-b6b5-f1aa2ec62703" path="/var/lib/kubelet/pods/ecbe850b-7c2f-4556-b6b5-f1aa2ec62703/volumes" Sep 10 23:55:36.356353 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.356313 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr"] Sep 10 23:55:36.356827 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.356754 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff709b92-c05f-41b6-8a5c-e0f59549ecd7" containerName="payload-processing" Sep 10 23:55:36.356827 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.356772 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff709b92-c05f-41b6-8a5c-e0f59549ecd7" containerName="payload-processing" Sep 10 23:55:36.356827 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.356788 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="492feec0-17ab-4a6c-b362-368a3853052d" containerName="payload-pre-processing" Sep 10 23:55:36.356827 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.356795 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="492feec0-17ab-4a6c-b362-368a3853052d" containerName="payload-pre-processing" Sep 10 23:55:36.356827 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.356807 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc" containerName="payload-processing" Sep 10 23:55:36.356827 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.356816 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc" containerName="payload-processing" Sep 10 23:55:36.356827 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.356830 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a673735-8cb8-4342-81a1-e7ec4357fbf7" containerName="istio-proxy" Sep 10 23:55:36.357209 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.356838 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a673735-8cb8-4342-81a1-e7ec4357fbf7" containerName="istio-proxy" Sep 10 23:55:36.357209 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.356847 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea312b45-cf18-4c8d-a7e6-10a5ef475ee4" containerName="payload-processing" Sep 10 23:55:36.357209 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.356855 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea312b45-cf18-4c8d-a7e6-10a5ef475ee4" containerName="payload-processing" Sep 10 23:55:36.357209 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.356867 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecbe850b-7c2f-4556-b6b5-f1aa2ec62703" containerName="payload-pre-processing" Sep 10 23:55:36.357209 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.356876 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecbe850b-7c2f-4556-b6b5-f1aa2ec62703" containerName="payload-pre-processing" Sep 10 23:55:36.357209 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.356893 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a673735-8cb8-4342-81a1-e7ec4357fbf7" containerName="istio-proxy" Sep 10 23:55:36.357209 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.356901 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a673735-8cb8-4342-81a1-e7ec4357fbf7" containerName="istio-proxy" Sep 10 23:55:36.357209 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.356911 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f42dfd8b-5bb6-4121-994a-0a5ff684470a" containerName="payload-pre-processing" Sep 10 23:55:36.357209 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.356920 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f42dfd8b-5bb6-4121-994a-0a5ff684470a" containerName="payload-pre-processing" Sep 10 23:55:36.357209 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.357017 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f42dfd8b-5bb6-4121-994a-0a5ff684470a" containerName="payload-pre-processing" Sep 10 23:55:36.357209 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.357029 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea312b45-cf18-4c8d-a7e6-10a5ef475ee4" containerName="payload-processing" Sep 10 23:55:36.357209 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.357041 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ea8d2ee-5861-4da6-8a12-f6cff2bbe0bc" containerName="payload-processing" Sep 10 23:55:36.357209 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.357052 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a673735-8cb8-4342-81a1-e7ec4357fbf7" containerName="istio-proxy" Sep 10 23:55:36.357209 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.357061 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a673735-8cb8-4342-81a1-e7ec4357fbf7" containerName="istio-proxy" Sep 10 23:55:36.357209 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.357070 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff709b92-c05f-41b6-8a5c-e0f59549ecd7" containerName="payload-processing" Sep 10 23:55:36.357209 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.357077 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="492feec0-17ab-4a6c-b362-368a3853052d" containerName="payload-pre-processing" Sep 10 23:55:36.357209 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.357085 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecbe850b-7c2f-4556-b6b5-f1aa2ec62703" containerName="payload-pre-processing" Sep 10 23:55:36.361662 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.361637 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.364495 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.364474 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-9eee5d-openshift-default-dockercfg-lx99d\"" Sep 10 23:55:36.382100 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.382073 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr"] Sep 10 23:55:36.522898 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.522867 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c820563-d145-4544-b705-374e0506d777-istio-podinfo\") pod \"e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.523102 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.522926 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-workload-socket\") pod \"e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.523102 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.522965 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c820563-d145-4544-b705-374e0506d777-istiod-ca-cert\") pod \"e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.523102 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.522992 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-credential-socket\") pod \"e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.523102 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.523008 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-workload-certs\") pod \"e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.523102 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.523048 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c820563-d145-4544-b705-374e0506d777-istio-token\") pod \"e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.523305 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.523145 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-istio-envoy\") pod \"e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.523305 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.523204 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-istio-data\") pod \"e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.523305 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.523230 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28dtd\" (UniqueName: \"kubernetes.io/projected/2c820563-d145-4544-b705-374e0506d777-kube-api-access-28dtd\") pod \"e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.624081 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.624004 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-workload-socket\") pod \"e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.624081 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.624036 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c820563-d145-4544-b705-374e0506d777-istiod-ca-cert\") pod \"e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.624081 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.624063 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-credential-socket\") pod \"e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.624346 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.624085 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-workload-certs\") pod \"e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.624346 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.624131 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c820563-d145-4544-b705-374e0506d777-istio-token\") pod \"e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.624346 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.624179 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-istio-envoy\") pod \"e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.624346 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.624221 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-istio-data\") pod \"e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.624346 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.624245 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28dtd\" (UniqueName: \"kubernetes.io/projected/2c820563-d145-4544-b705-374e0506d777-kube-api-access-28dtd\") pod \"e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.624346 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.624277 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c820563-d145-4544-b705-374e0506d777-istio-podinfo\") pod \"e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.624655 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.624493 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-workload-socket\") pod \"e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.624714 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.624672 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-workload-certs\") pod \"e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.624753 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.624724 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-istio-data\") pod \"e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.624753 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.624739 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-credential-socket\") pod \"e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.624972 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.624925 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c820563-d145-4544-b705-374e0506d777-istiod-ca-cert\") pod \"e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.626483 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.626462 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-istio-envoy\") pod \"e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.626782 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.626763 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c820563-d145-4544-b705-374e0506d777-istio-podinfo\") pod \"e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.632036 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.632012 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c820563-d145-4544-b705-374e0506d777-istio-token\") pod \"e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.632126 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.632100 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28dtd\" (UniqueName: \"kubernetes.io/projected/2c820563-d145-4544-b705-374e0506d777-kube-api-access-28dtd\") pod \"e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.673643 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.673622 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:36.951908 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.951861 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr"] Sep 10 23:55:36.955978 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:55:36.955947 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c820563_d145_4544_b705_374e0506d777.slice/crio-8a2f39a2cf5f91622061061e339874193a04c0c4f5a917467bb301ad2e94ec26 WatchSource:0}: Error finding container 8a2f39a2cf5f91622061061e339874193a04c0c4f5a917467bb301ad2e94ec26: Status 404 returned error can't find the container with id 8a2f39a2cf5f91622061061e339874193a04c0c4f5a917467bb301ad2e94ec26 Sep 10 23:55:36.958157 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.958121 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:55:36.958266 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.958190 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:55:36.958266 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:36.958218 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:55:37.076168 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:37.076132 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" event={"ID":"2c820563-d145-4544-b705-374e0506d777","Type":"ContainerStarted","Data":"bc266207901f898d579e81ee246e79b2ba3445e653b9d6a2555a8be80ee0d9f2"} Sep 10 23:55:37.076329 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:37.076207 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" event={"ID":"2c820563-d145-4544-b705-374e0506d777","Type":"ContainerStarted","Data":"8a2f39a2cf5f91622061061e339874193a04c0c4f5a917467bb301ad2e94ec26"} Sep 10 23:55:37.107300 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:37.107244 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" podStartSLOduration=1.107224765 podStartE2EDuration="1.107224765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:37.105805051 +0000 UTC m=+1152.573279400" watchObservedRunningTime="2026-09-10 23:55:37.107224765 +0000 UTC m=+1152.574699089" Sep 10 23:55:37.674224 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:37.674190 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:38.679834 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:38.679804 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:39.085626 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:39.085538 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:39.086813 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:39.086788 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:41.012456 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:41.012419 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7"] Sep 10 23:55:41.015923 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:41.015905 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7" Sep 10 23:55:41.018491 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:41.018439 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-9eee5d\"" Sep 10 23:55:41.018491 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:41.018490 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-9eee5d-dockercfg-5kf9g\"" Sep 10 23:55:41.027505 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:41.027483 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7"] Sep 10 23:55:41.080343 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:41.080314 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58"] Sep 10 23:55:41.083472 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:41.083453 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58" Sep 10 23:55:41.096240 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:41.096217 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58"] Sep 10 23:55:41.164624 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:41.164589 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd46b037-bcc7-41b5-90c7-a343097ca4d2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7\" (UID: \"dd46b037-bcc7-41b5-90c7-a343097ca4d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7" Sep 10 23:55:41.164784 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:41.164628 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrqlt\" (UniqueName: \"kubernetes.io/projected/920003d3-ad43-4a7a-9bff-8cf3a0cd222c-kube-api-access-vrqlt\") pod \"payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58\" (UID: \"920003d3-ad43-4a7a-9bff-8cf3a0cd222c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58" Sep 10 23:55:41.164826 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:41.164799 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvq5j\" (UniqueName: \"kubernetes.io/projected/dd46b037-bcc7-41b5-90c7-a343097ca4d2-kube-api-access-vvq5j\") pod \"payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7\" (UID: \"dd46b037-bcc7-41b5-90c7-a343097ca4d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7" Sep 10 23:55:41.164864 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:41.164837 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/920003d3-ad43-4a7a-9bff-8cf3a0cd222c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58\" (UID: \"920003d3-ad43-4a7a-9bff-8cf3a0cd222c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58" Sep 10 23:55:41.266010 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:41.265894 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd46b037-bcc7-41b5-90c7-a343097ca4d2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7\" (UID: \"dd46b037-bcc7-41b5-90c7-a343097ca4d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7" Sep 10 23:55:41.266010 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:41.265976 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrqlt\" (UniqueName: \"kubernetes.io/projected/920003d3-ad43-4a7a-9bff-8cf3a0cd222c-kube-api-access-vrqlt\") pod \"payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58\" (UID: \"920003d3-ad43-4a7a-9bff-8cf3a0cd222c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58" Sep 10 23:55:41.266233 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:41.266024 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvq5j\" (UniqueName: \"kubernetes.io/projected/dd46b037-bcc7-41b5-90c7-a343097ca4d2-kube-api-access-vvq5j\") pod \"payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7\" (UID: \"dd46b037-bcc7-41b5-90c7-a343097ca4d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7" Sep 10 23:55:41.266233 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:41.266070 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/920003d3-ad43-4a7a-9bff-8cf3a0cd222c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58\" (UID: \"920003d3-ad43-4a7a-9bff-8cf3a0cd222c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58" Sep 10 23:55:41.266821 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:41.266796 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/920003d3-ad43-4a7a-9bff-8cf3a0cd222c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58\" (UID: \"920003d3-ad43-4a7a-9bff-8cf3a0cd222c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58" Sep 10 23:55:41.266924 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:41.266800 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd46b037-bcc7-41b5-90c7-a343097ca4d2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7\" (UID: \"dd46b037-bcc7-41b5-90c7-a343097ca4d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7" Sep 10 23:55:41.292092 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:41.292063 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrqlt\" (UniqueName: \"kubernetes.io/projected/920003d3-ad43-4a7a-9bff-8cf3a0cd222c-kube-api-access-vrqlt\") pod \"payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58\" (UID: \"920003d3-ad43-4a7a-9bff-8cf3a0cd222c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58" Sep 10 23:55:41.292483 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:41.292467 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvq5j\" (UniqueName: \"kubernetes.io/projected/dd46b037-bcc7-41b5-90c7-a343097ca4d2-kube-api-access-vvq5j\") pod \"payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7\" (UID: \"dd46b037-bcc7-41b5-90c7-a343097ca4d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7" Sep 10 23:55:41.326696 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:41.326659 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7" Sep 10 23:55:41.394715 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:41.394473 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58" Sep 10 23:55:41.618587 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:41.618562 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7"] Sep 10 23:55:41.620080 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:55:41.620044 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd46b037_bcc7_41b5_90c7_a343097ca4d2.slice/crio-eec98cca4c8b11b03ff4e6affc81bd191f1521116f62e35bc752fd12c5a8cd03 WatchSource:0}: Error finding container eec98cca4c8b11b03ff4e6affc81bd191f1521116f62e35bc752fd12c5a8cd03: Status 404 returned error can't find the container with id eec98cca4c8b11b03ff4e6affc81bd191f1521116f62e35bc752fd12c5a8cd03 Sep 10 23:55:41.702272 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:41.702180 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58"] Sep 10 23:55:41.704273 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:55:41.704241 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod920003d3_ad43_4a7a_9bff_8cf3a0cd222c.slice/crio-5dc0ee4796d2f796542eeeb3f6d1494abc11b41346228ad7f0b0cf3fce77dc0b WatchSource:0}: Error finding container 5dc0ee4796d2f796542eeeb3f6d1494abc11b41346228ad7f0b0cf3fce77dc0b: Status 404 returned error can't find the container with id 5dc0ee4796d2f796542eeeb3f6d1494abc11b41346228ad7f0b0cf3fce77dc0b Sep 10 23:55:42.101747 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.101704 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7" event={"ID":"dd46b037-bcc7-41b5-90c7-a343097ca4d2","Type":"ContainerStarted","Data":"d836cb5114fafde9f00cfa5ee4bc22fc964def850b9086b8e314c5d759d8bcd7"} Sep 10 23:55:42.102155 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.101772 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7" event={"ID":"dd46b037-bcc7-41b5-90c7-a343097ca4d2","Type":"ContainerStarted","Data":"eec98cca4c8b11b03ff4e6affc81bd191f1521116f62e35bc752fd12c5a8cd03"} Sep 10 23:55:42.102155 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.101801 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7" Sep 10 23:55:42.103360 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.103335 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58" event={"ID":"920003d3-ad43-4a7a-9bff-8cf3a0cd222c","Type":"ContainerStarted","Data":"f9bd76587e89fa6770a815c4a26df53515531cde5911a2ce47990d19c398e636"} Sep 10 23:55:42.103482 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.103387 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58" Sep 10 23:55:42.103482 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.103410 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58" event={"ID":"920003d3-ad43-4a7a-9bff-8cf3a0cd222c","Type":"ContainerStarted","Data":"5dc0ee4796d2f796542eeeb3f6d1494abc11b41346228ad7f0b0cf3fce77dc0b"} Sep 10 23:55:42.132448 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.132399 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7" podStartSLOduration=2.132382608 podStartE2EDuration="2.132382608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:42.13136317 +0000 UTC m=+1157.598837522" watchObservedRunningTime="2026-09-10 23:55:42.132382608 +0000 UTC m=+1157.599857226" Sep 10 23:55:42.175693 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.175640 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58" podStartSLOduration=1.175623349 podStartE2EDuration="1.175623349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:42.174038209 +0000 UTC m=+1157.641512535" watchObservedRunningTime="2026-09-10 23:55:42.175623349 +0000 UTC m=+1157.643097674" Sep 10 23:55:42.251061 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.251028 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k"] Sep 10 23:55:42.254538 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.254504 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k" Sep 10 23:55:42.257064 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.257037 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-37692b\"" Sep 10 23:55:42.257206 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.257170 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-37692b-dockercfg-whwk5\"" Sep 10 23:55:42.269029 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.268920 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k"] Sep 10 23:55:42.284655 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.284628 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd"] Sep 10 23:55:42.287903 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.287883 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd" Sep 10 23:55:42.301889 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.301868 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd"] Sep 10 23:55:42.378743 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.378714 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d53a069-c3ba-4963-b577-fd3035240d4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd\" (UID: \"8d53a069-c3ba-4963-b577-fd3035240d4e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd" Sep 10 23:55:42.378928 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.378755 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdnkl\" (UniqueName: \"kubernetes.io/projected/8d53a069-c3ba-4963-b577-fd3035240d4e-kube-api-access-vdnkl\") pod \"payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd\" (UID: \"8d53a069-c3ba-4963-b577-fd3035240d4e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd" Sep 10 23:55:42.378928 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.378871 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq25b\" (UniqueName: \"kubernetes.io/projected/3e0c579f-83f1-4f7a-b52f-a71d18df718a-kube-api-access-vq25b\") pod \"payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k\" (UID: \"3e0c579f-83f1-4f7a-b52f-a71d18df718a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k" Sep 10 23:55:42.379065 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.378926 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e0c579f-83f1-4f7a-b52f-a71d18df718a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k\" (UID: \"3e0c579f-83f1-4f7a-b52f-a71d18df718a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k" Sep 10 23:55:42.479755 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.479725 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d53a069-c3ba-4963-b577-fd3035240d4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd\" (UID: \"8d53a069-c3ba-4963-b577-fd3035240d4e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd" Sep 10 23:55:42.479926 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.479766 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdnkl\" (UniqueName: \"kubernetes.io/projected/8d53a069-c3ba-4963-b577-fd3035240d4e-kube-api-access-vdnkl\") pod \"payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd\" (UID: \"8d53a069-c3ba-4963-b577-fd3035240d4e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd" Sep 10 23:55:42.479926 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.479817 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq25b\" (UniqueName: \"kubernetes.io/projected/3e0c579f-83f1-4f7a-b52f-a71d18df718a-kube-api-access-vq25b\") pod \"payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k\" (UID: \"3e0c579f-83f1-4f7a-b52f-a71d18df718a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k" Sep 10 23:55:42.479926 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.479843 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e0c579f-83f1-4f7a-b52f-a71d18df718a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k\" (UID: \"3e0c579f-83f1-4f7a-b52f-a71d18df718a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k" Sep 10 23:55:42.480552 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.480531 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e0c579f-83f1-4f7a-b52f-a71d18df718a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k\" (UID: \"3e0c579f-83f1-4f7a-b52f-a71d18df718a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k" Sep 10 23:55:42.480628 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.480605 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d53a069-c3ba-4963-b577-fd3035240d4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd\" (UID: \"8d53a069-c3ba-4963-b577-fd3035240d4e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd" Sep 10 23:55:42.490300 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.490279 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq25b\" (UniqueName: \"kubernetes.io/projected/3e0c579f-83f1-4f7a-b52f-a71d18df718a-kube-api-access-vq25b\") pod \"payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k\" (UID: \"3e0c579f-83f1-4f7a-b52f-a71d18df718a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k" Sep 10 23:55:42.490413 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.490346 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdnkl\" (UniqueName: \"kubernetes.io/projected/8d53a069-c3ba-4963-b577-fd3035240d4e-kube-api-access-vdnkl\") pod \"payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd\" (UID: \"8d53a069-c3ba-4963-b577-fd3035240d4e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd" Sep 10 23:55:42.568130 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.568103 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k" Sep 10 23:55:42.610853 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.610809 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd" Sep 10 23:55:42.853765 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.853735 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k"] Sep 10 23:55:42.854842 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:55:42.854808 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e0c579f_83f1_4f7a_b52f_a71d18df718a.slice/crio-3db16b4377435307bcc1a9261d43d773fb17ffc70190cbaf6598569e5b5ae10d WatchSource:0}: Error finding container 3db16b4377435307bcc1a9261d43d773fb17ffc70190cbaf6598569e5b5ae10d: Status 404 returned error can't find the container with id 3db16b4377435307bcc1a9261d43d773fb17ffc70190cbaf6598569e5b5ae10d Sep 10 23:55:42.892882 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:42.892857 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd"] Sep 10 23:55:42.893576 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:55:42.893546 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d53a069_c3ba_4963_b577_fd3035240d4e.slice/crio-4d9cfe6f05994124887993d5e88a2b8d1804ecd4435e8532f56ea4de156dc0ca WatchSource:0}: Error finding container 4d9cfe6f05994124887993d5e88a2b8d1804ecd4435e8532f56ea4de156dc0ca: Status 404 returned error can't find the container with id 4d9cfe6f05994124887993d5e88a2b8d1804ecd4435e8532f56ea4de156dc0ca Sep 10 23:55:43.110397 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:43.110366 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k" event={"ID":"3e0c579f-83f1-4f7a-b52f-a71d18df718a","Type":"ContainerStarted","Data":"34b1cf77906f5d84d8b720095071470ec89204586d77995b8920fb28c3284231"} Sep 10 23:55:43.110858 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:43.110420 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k" event={"ID":"3e0c579f-83f1-4f7a-b52f-a71d18df718a","Type":"ContainerStarted","Data":"3db16b4377435307bcc1a9261d43d773fb17ffc70190cbaf6598569e5b5ae10d"} Sep 10 23:55:43.110858 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:43.110509 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k" Sep 10 23:55:43.117300 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:43.117257 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd" event={"ID":"8d53a069-c3ba-4963-b577-fd3035240d4e","Type":"ContainerStarted","Data":"b3b1001d7dd45ed272552548abb05786d446c6dafbdea89be3bb7d216b2d1c53"} Sep 10 23:55:43.117458 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:43.117354 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd" event={"ID":"8d53a069-c3ba-4963-b577-fd3035240d4e","Type":"ContainerStarted","Data":"4d9cfe6f05994124887993d5e88a2b8d1804ecd4435e8532f56ea4de156dc0ca"} Sep 10 23:55:43.117528 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:43.117468 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd" Sep 10 23:55:43.129172 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:43.129128 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k" podStartSLOduration=1.129115216 podStartE2EDuration="1.129115216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:43.12858306 +0000 UTC m=+1158.596057387" watchObservedRunningTime="2026-09-10 23:55:43.129115216 +0000 UTC m=+1158.596589585" Sep 10 23:55:43.146497 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:43.146401 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd" podStartSLOduration=1.146383756 podStartE2EDuration="1.146383756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:43.144334435 +0000 UTC m=+1158.611808761" watchObservedRunningTime="2026-09-10 23:55:43.146383756 +0000 UTC m=+1158.613858084" Sep 10 23:55:53.119324 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:53.119291 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7" Sep 10 23:55:53.119776 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:53.119361 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58" Sep 10 23:55:54.123425 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:54.123394 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k" Sep 10 23:55:54.123772 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:54.123462 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd" Sep 10 23:55:55.128116 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:55.128073 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 23:55:55.128496 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:55.128182 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" podUID="2c820563-d145-4544-b705-374e0506d777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 23:55:56.188246 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:56.188208 2578 generic.go:356] "Generic (PLEG): container finished" podID="2c820563-d145-4544-b705-374e0506d777" containerID="bc266207901f898d579e81ee246e79b2ba3445e653b9d6a2555a8be80ee0d9f2" exitCode=0 Sep 10 23:55:56.188671 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:56.188282 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" event={"ID":"2c820563-d145-4544-b705-374e0506d777","Type":"ContainerDied","Data":"bc266207901f898d579e81ee246e79b2ba3445e653b9d6a2555a8be80ee0d9f2"} Sep 10 23:55:56.188671 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:56.188658 2578 scope.go:117] "RemoveContainer" containerID="bc266207901f898d579e81ee246e79b2ba3445e653b9d6a2555a8be80ee0d9f2" Sep 10 23:55:56.189651 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:56.189621 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:55:56.189755 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:56.189707 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:55:56.189822 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:56.189753 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 23:55:57.194967 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:57.194918 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" event={"ID":"2c820563-d145-4544-b705-374e0506d777","Type":"ContainerStarted","Data":"e2fac45351d08b7644de89743a7f3f6ed1c20c690233a4e95f834ca43ea67daf"} Sep 10 23:55:57.674691 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:57.674651 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:55:57.674927 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:57.674904 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Sep 10 23:55:57.675006 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:57.674985 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" podUID="2c820563-d145-4544-b705-374e0506d777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Sep 10 23:55:58.674551 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:58.674518 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Sep 10 23:55:58.674962 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:58.674572 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" podUID="2c820563-d145-4544-b705-374e0506d777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Sep 10 23:55:59.674482 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:59.674444 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Sep 10 23:55:59.674687 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:55:59.674502 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" podUID="2c820563-d145-4544-b705-374e0506d777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Sep 10 23:56:00.674585 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:00.674546 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Sep 10 23:56:00.674761 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:00.674604 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" podUID="2c820563-d145-4544-b705-374e0506d777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Sep 10 23:56:01.674492 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:01.674457 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Sep 10 23:56:01.674688 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:01.674518 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" podUID="2c820563-d145-4544-b705-374e0506d777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Sep 10 23:56:02.674217 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:02.674182 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Sep 10 23:56:02.674613 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:02.674249 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" podUID="2c820563-d145-4544-b705-374e0506d777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Sep 10 23:56:03.678358 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:03.678319 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:56:03.678731 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:03.678574 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:56:03.679167 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:03.679148 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 10 23:56:10.195875 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:10.195831 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds"] Sep 10 23:56:10.206518 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:10.206472 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds" Sep 10 23:56:10.208884 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:10.208861 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds"] Sep 10 23:56:10.209479 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:10.209461 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-dd5958-dockercfg-bnbdr\"" Sep 10 23:56:10.209829 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:10.209811 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-dd5958\"" Sep 10 23:56:10.228259 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:10.228234 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn"] Sep 10 23:56:10.230813 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:10.230796 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn" Sep 10 23:56:10.241169 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:10.241141 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn"] Sep 10 23:56:10.337761 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:10.337727 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a775eaba-4ecf-40f7-9190-4d7ada5a92ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds\" (UID: \"a775eaba-4ecf-40f7-9190-4d7ada5a92ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds" Sep 10 23:56:10.337761 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:10.337764 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t45rc\" (UniqueName: \"kubernetes.io/projected/20688dce-b817-4241-880a-e61c4e240755-kube-api-access-t45rc\") pod \"payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn\" (UID: \"20688dce-b817-4241-880a-e61c4e240755\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn" Sep 10 23:56:10.338018 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:10.337787 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20688dce-b817-4241-880a-e61c4e240755-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn\" (UID: \"20688dce-b817-4241-880a-e61c4e240755\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn" Sep 10 23:56:10.338018 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:10.337816 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz89k\" (UniqueName: \"kubernetes.io/projected/a775eaba-4ecf-40f7-9190-4d7ada5a92ab-kube-api-access-pz89k\") pod \"payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds\" (UID: \"a775eaba-4ecf-40f7-9190-4d7ada5a92ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds" Sep 10 23:56:10.438596 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:10.438562 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a775eaba-4ecf-40f7-9190-4d7ada5a92ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds\" (UID: \"a775eaba-4ecf-40f7-9190-4d7ada5a92ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds" Sep 10 23:56:10.438801 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:10.438604 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t45rc\" (UniqueName: \"kubernetes.io/projected/20688dce-b817-4241-880a-e61c4e240755-kube-api-access-t45rc\") pod \"payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn\" (UID: \"20688dce-b817-4241-880a-e61c4e240755\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn" Sep 10 23:56:10.438801 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:10.438626 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20688dce-b817-4241-880a-e61c4e240755-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn\" (UID: \"20688dce-b817-4241-880a-e61c4e240755\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn" Sep 10 23:56:10.438801 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:10.438663 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz89k\" (UniqueName: \"kubernetes.io/projected/a775eaba-4ecf-40f7-9190-4d7ada5a92ab-kube-api-access-pz89k\") pod \"payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds\" (UID: \"a775eaba-4ecf-40f7-9190-4d7ada5a92ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds" Sep 10 23:56:10.439553 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:10.439525 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a775eaba-4ecf-40f7-9190-4d7ada5a92ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds\" (UID: \"a775eaba-4ecf-40f7-9190-4d7ada5a92ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds" Sep 10 23:56:10.439647 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:10.439525 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20688dce-b817-4241-880a-e61c4e240755-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn\" (UID: \"20688dce-b817-4241-880a-e61c4e240755\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn" Sep 10 23:56:10.446878 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:10.446824 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t45rc\" (UniqueName: \"kubernetes.io/projected/20688dce-b817-4241-880a-e61c4e240755-kube-api-access-t45rc\") pod \"payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn\" (UID: \"20688dce-b817-4241-880a-e61c4e240755\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn" Sep 10 23:56:10.447314 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:10.447293 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz89k\" (UniqueName: \"kubernetes.io/projected/a775eaba-4ecf-40f7-9190-4d7ada5a92ab-kube-api-access-pz89k\") pod \"payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds\" (UID: \"a775eaba-4ecf-40f7-9190-4d7ada5a92ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds" Sep 10 23:56:10.518853 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:10.518825 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds" Sep 10 23:56:10.542262 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:10.542225 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn" Sep 10 23:56:10.796327 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:10.796303 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds"] Sep 10 23:56:10.798148 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:56:10.798124 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda775eaba_4ecf_40f7_9190_4d7ada5a92ab.slice/crio-d2240e175519bb596b0cad90fd1b3e07ef21eadfaa62bc1ae0a2ab9e72d9ec23 WatchSource:0}: Error finding container d2240e175519bb596b0cad90fd1b3e07ef21eadfaa62bc1ae0a2ab9e72d9ec23: Status 404 returned error can't find the container with id d2240e175519bb596b0cad90fd1b3e07ef21eadfaa62bc1ae0a2ab9e72d9ec23 Sep 10 23:56:10.823194 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:10.823163 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn"] Sep 10 23:56:10.828993 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:56:10.828964 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20688dce_b817_4241_880a_e61c4e240755.slice/crio-d8bb1ff2793ee3538daa88309c3478e109f23c96ba5d0ed1c996a0f0ae020160 WatchSource:0}: Error finding container d8bb1ff2793ee3538daa88309c3478e109f23c96ba5d0ed1c996a0f0ae020160: Status 404 returned error can't find the container with id d8bb1ff2793ee3538daa88309c3478e109f23c96ba5d0ed1c996a0f0ae020160 Sep 10 23:56:11.260518 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:11.260485 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn" event={"ID":"20688dce-b817-4241-880a-e61c4e240755","Type":"ContainerStarted","Data":"772cd92b085346354f29a0259c8ac13ffbf8653da489ed723446a9e8c54e67a5"} Sep 10 23:56:11.261000 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:11.260545 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn" event={"ID":"20688dce-b817-4241-880a-e61c4e240755","Type":"ContainerStarted","Data":"d8bb1ff2793ee3538daa88309c3478e109f23c96ba5d0ed1c996a0f0ae020160"} Sep 10 23:56:11.261000 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:11.260650 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn" Sep 10 23:56:11.262345 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:11.262319 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds" event={"ID":"a775eaba-4ecf-40f7-9190-4d7ada5a92ab","Type":"ContainerStarted","Data":"58bd636540a58b330e042ecf798605b7e92781ef113a11b676592bfdeed9d143"} Sep 10 23:56:11.262468 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:11.262360 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds" event={"ID":"a775eaba-4ecf-40f7-9190-4d7ada5a92ab","Type":"ContainerStarted","Data":"d2240e175519bb596b0cad90fd1b3e07ef21eadfaa62bc1ae0a2ab9e72d9ec23"} Sep 10 23:56:11.262468 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:11.262379 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds" Sep 10 23:56:11.276639 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:11.276585 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn" podStartSLOduration=1.276566407 podStartE2EDuration="1.276566407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:56:11.275238719 +0000 UTC m=+1186.742713046" watchObservedRunningTime="2026-09-10 23:56:11.276566407 +0000 UTC m=+1186.744040735" Sep 10 23:56:11.292603 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:11.292554 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds" podStartSLOduration=1.2925417719999999 podStartE2EDuration="1.292541772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:56:11.2913609 +0000 UTC m=+1186.758835230" watchObservedRunningTime="2026-09-10 23:56:11.292541772 +0000 UTC m=+1186.760016096" Sep 10 23:56:11.802771 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:11.802736 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw"] Sep 10 23:56:11.805983 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:11.805963 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw" Sep 10 23:56:11.808439 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:11.808416 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-e96d4c-dockercfg-mltkh\"" Sep 10 23:56:11.808551 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:11.808449 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-e96d4c\"" Sep 10 23:56:11.815759 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:11.815730 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw"] Sep 10 23:56:11.855332 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:11.855279 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwnjh\" (UniqueName: \"kubernetes.io/projected/191bd9a2-0920-4f72-9122-bc39ed6f7319-kube-api-access-zwnjh\") pod \"payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw\" (UID: \"191bd9a2-0920-4f72-9122-bc39ed6f7319\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw" Sep 10 23:56:11.855332 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:11.855319 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/191bd9a2-0920-4f72-9122-bc39ed6f7319-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw\" (UID: \"191bd9a2-0920-4f72-9122-bc39ed6f7319\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw" Sep 10 23:56:11.956302 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:11.956271 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwnjh\" (UniqueName: \"kubernetes.io/projected/191bd9a2-0920-4f72-9122-bc39ed6f7319-kube-api-access-zwnjh\") pod \"payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw\" (UID: \"191bd9a2-0920-4f72-9122-bc39ed6f7319\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw" Sep 10 23:56:11.956499 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:11.956311 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/191bd9a2-0920-4f72-9122-bc39ed6f7319-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw\" (UID: \"191bd9a2-0920-4f72-9122-bc39ed6f7319\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw" Sep 10 23:56:11.956975 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:11.956955 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/191bd9a2-0920-4f72-9122-bc39ed6f7319-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw\" (UID: \"191bd9a2-0920-4f72-9122-bc39ed6f7319\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw" Sep 10 23:56:11.964190 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:11.964169 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwnjh\" (UniqueName: \"kubernetes.io/projected/191bd9a2-0920-4f72-9122-bc39ed6f7319-kube-api-access-zwnjh\") pod \"payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw\" (UID: \"191bd9a2-0920-4f72-9122-bc39ed6f7319\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw" Sep 10 23:56:12.117596 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:12.117509 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw" Sep 10 23:56:12.391587 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:12.391563 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw"] Sep 10 23:56:12.393604 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:56:12.393580 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191bd9a2_0920_4f72_9122_bc39ed6f7319.slice/crio-d4504c3e19ba164cf14e6811a91bc24d5d2724db859913931d9634d33214785b WatchSource:0}: Error finding container d4504c3e19ba164cf14e6811a91bc24d5d2724db859913931d9634d33214785b: Status 404 returned error can't find the container with id d4504c3e19ba164cf14e6811a91bc24d5d2724db859913931d9634d33214785b Sep 10 23:56:13.274098 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:13.274062 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw" event={"ID":"191bd9a2-0920-4f72-9122-bc39ed6f7319","Type":"ContainerStarted","Data":"ea196fdeffbdf7b1e1ec936f8e17fab7b50bd765b1a906ebbcb29ebe2b707464"} Sep 10 23:56:13.274347 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:13.274126 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw" event={"ID":"191bd9a2-0920-4f72-9122-bc39ed6f7319","Type":"ContainerStarted","Data":"d4504c3e19ba164cf14e6811a91bc24d5d2724db859913931d9634d33214785b"} Sep 10 23:56:13.274347 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:13.274159 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw" Sep 10 23:56:13.292124 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:13.292073 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw" podStartSLOduration=2.292058542 podStartE2EDuration="2.292058542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:56:13.290245141 +0000 UTC m=+1188.757719488" watchObservedRunningTime="2026-09-10 23:56:13.292058542 +0000 UTC m=+1188.759532867" Sep 10 23:56:22.270352 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:22.270324 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds" Sep 10 23:56:22.270770 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:22.270537 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn" Sep 10 23:56:24.282439 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:24.282405 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw" Sep 10 23:56:25.038157 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:25.038124 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zkr6f_aacd921b-5c15-4f7f-8685-cfceebfe5407/console-operator/2.log" Sep 10 23:56:25.039796 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:56:25.039771 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zkr6f_aacd921b-5c15-4f7f-8685-cfceebfe5407/console-operator/2.log" Sep 10 23:57:25.112220 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:57:25.112155 2578 scope.go:117] "RemoveContainer" containerID="df254a9beccd3bf25e1ee41fe7facab558cc0d249fd5cb0e71f9adb336627902" Sep 10 23:58:25.355192 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:58:25.355154 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c869d5a-cqw2d"] Sep 10 23:58:25.358394 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:58:25.358377 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c869d5a-cqw2d" Sep 10 23:58:25.360824 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:58:25.360801 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-22lrj\"" Sep 10 23:58:25.360963 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:58:25.360806 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 23:58:25.364960 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:58:25.364921 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c869d5a-cqw2d"] Sep 10 23:58:25.519059 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:58:25.519028 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db61f285-925c-4be4-b4da-3dd4965d6b88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6c869d5a-cqw2d\" (UID: \"db61f285-925c-4be4-b4da-3dd4965d6b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c869d5a-cqw2d" Sep 10 23:58:25.620234 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:58:25.620147 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db61f285-925c-4be4-b4da-3dd4965d6b88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6c869d5a-cqw2d\" (UID: \"db61f285-925c-4be4-b4da-3dd4965d6b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c869d5a-cqw2d" Sep 10 23:58:25.620771 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:58:25.620748 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db61f285-925c-4be4-b4da-3dd4965d6b88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6c869d5a-cqw2d\" (UID: \"db61f285-925c-4be4-b4da-3dd4965d6b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c869d5a-cqw2d" Sep 10 23:58:25.670586 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:58:25.670554 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c869d5a-cqw2d" Sep 10 23:58:25.942040 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:58:25.942005 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c869d5a-cqw2d"] Sep 10 23:58:25.945750 ip-10-0-139-142 kubenswrapper[2578]: W0910 23:58:25.945722 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb61f285_925c_4be4_b4da_3dd4965d6b88.slice/crio-b31eaa3a4a2c73ca7d3f196f6c87ded3752ff78d9af56504db709351d41c3555 WatchSource:0}: Error finding container b31eaa3a4a2c73ca7d3f196f6c87ded3752ff78d9af56504db709351d41c3555: Status 404 returned error can't find the container with id b31eaa3a4a2c73ca7d3f196f6c87ded3752ff78d9af56504db709351d41c3555 Sep 10 23:58:26.913690 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:58:26.913597 2578 generic.go:356] "Generic (PLEG): container finished" podID="db61f285-925c-4be4-b4da-3dd4965d6b88" containerID="b84a3a825231264fe31ad99127c3751a3d05f2b9d3bcc4d206fe8bb9ebebfffc" exitCode=0 Sep 10 23:58:26.913690 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:58:26.913660 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c869d5a-cqw2d" event={"ID":"db61f285-925c-4be4-b4da-3dd4965d6b88","Type":"ContainerDied","Data":"b84a3a825231264fe31ad99127c3751a3d05f2b9d3bcc4d206fe8bb9ebebfffc"} Sep 10 23:58:26.914120 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:58:26.913730 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c869d5a-cqw2d" event={"ID":"db61f285-925c-4be4-b4da-3dd4965d6b88","Type":"ContainerStarted","Data":"b31eaa3a4a2c73ca7d3f196f6c87ded3752ff78d9af56504db709351d41c3555"} Sep 10 23:58:28.056353 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:58:28.056324 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c869d5a-cqw2d" Sep 10 23:58:28.242958 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:58:28.242846 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db61f285-925c-4be4-b4da-3dd4965d6b88-maas-api-service-ca\") pod \"db61f285-925c-4be4-b4da-3dd4965d6b88\" (UID: \"db61f285-925c-4be4-b4da-3dd4965d6b88\") " Sep 10 23:58:28.243230 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:58:28.243207 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db61f285-925c-4be4-b4da-3dd4965d6b88-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "db61f285-925c-4be4-b4da-3dd4965d6b88" (UID: "db61f285-925c-4be4-b4da-3dd4965d6b88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:58:28.343801 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:58:28.343767 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db61f285-925c-4be4-b4da-3dd4965d6b88-maas-api-service-ca\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 10 23:58:28.925517 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:58:28.925490 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c869d5a-cqw2d" Sep 10 23:58:28.925721 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:58:28.925490 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c869d5a-cqw2d" event={"ID":"db61f285-925c-4be4-b4da-3dd4965d6b88","Type":"ContainerDied","Data":"b31eaa3a4a2c73ca7d3f196f6c87ded3752ff78d9af56504db709351d41c3555"} Sep 10 23:58:28.925721 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:58:28.925614 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b31eaa3a4a2c73ca7d3f196f6c87ded3752ff78d9af56504db709351d41c3555" Sep 10 23:58:41.127194 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:58:41.127163 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c869d5a-cqw2d"] Sep 10 23:58:41.133281 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:58:41.133251 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6c869d5a-cqw2d"] Sep 10 23:58:43.044520 ip-10-0-139-142 kubenswrapper[2578]: I0910 23:58:43.044487 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db61f285-925c-4be4-b4da-3dd4965d6b88" path="/var/lib/kubelet/pods/db61f285-925c-4be4-b4da-3dd4965d6b88/volumes" Sep 11 00:00:00.160042 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:00.160002 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-37692b-29818080-vgfhv"] Sep 11 00:00:00.160549 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:00.160385 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db61f285-925c-4be4-b4da-3dd4965d6b88" containerName="revoke-keys" Sep 11 00:00:00.160549 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:00.160398 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="db61f285-925c-4be4-b4da-3dd4965d6b88" containerName="revoke-keys" Sep 11 00:00:00.160549 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:00.160463 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="db61f285-925c-4be4-b4da-3dd4965d6b88" containerName="revoke-keys" Sep 11 00:00:00.163330 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:00.163301 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9eee5d-29818080-xp24x"] Sep 11 00:00:00.163450 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:00.163422 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-37692b-29818080-vgfhv" Sep 11 00:00:00.165982 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:00.165961 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-22lrj\"" Sep 11 00:00:00.167464 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:00.167448 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9eee5d-29818080-xp24x" Sep 11 00:00:00.188177 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:00.188152 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-37692b-29818080-vgfhv" Sep 11 00:00:00.196544 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:00.196524 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9eee5d-29818080-xp24x" Sep 11 00:00:00.221769 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:00.221729 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9eee5d-29818080-xp24x"] Sep 11 00:00:00.222340 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:00.222314 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-37692b-29818080-vgfhv"] Sep 11 00:00:00.495269 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:00.495191 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-37692b-29818080-vgfhv"] Sep 11 00:00:00.498362 ip-10-0-139-142 kubenswrapper[2578]: W0911 00:00:00.498329 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76dd3efc_c95c_48d4_9910_febb821f1260.slice/crio-8e49c9122addd006ac6c6e55b470f0f32628c7cd33bc5fa9214192e42564a762 WatchSource:0}: Error finding container 8e49c9122addd006ac6c6e55b470f0f32628c7cd33bc5fa9214192e42564a762: Status 404 returned error can't find the container with id 8e49c9122addd006ac6c6e55b470f0f32628c7cd33bc5fa9214192e42564a762 Sep 11 00:00:00.517207 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:00.517177 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9eee5d-29818080-xp24x"] Sep 11 00:00:00.518325 ip-10-0-139-142 kubenswrapper[2578]: W0911 00:00:00.518301 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f9dae9_20bd_4ab6_b2d9_a103017f8087.slice/crio-52f529acfa58647e05ee229a4ce6e1e8b409f588cd7e298b5e1b4109e698403b WatchSource:0}: Error finding container 52f529acfa58647e05ee229a4ce6e1e8b409f588cd7e298b5e1b4109e698403b: Status 404 returned error can't find the container with id 52f529acfa58647e05ee229a4ce6e1e8b409f588cd7e298b5e1b4109e698403b Sep 11 00:00:01.368568 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:01.368532 2578 generic.go:356] "Generic (PLEG): container finished" podID="76dd3efc-c95c-48d4-9910-febb821f1260" containerID="e827a7223cf119a4ed20a14be44ee6723ac8c0eaa33a520d24abe089cb6212cd" exitCode=0 Sep 11 00:00:01.369004 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:01.368610 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-37692b-29818080-vgfhv" event={"ID":"76dd3efc-c95c-48d4-9910-febb821f1260","Type":"ContainerDied","Data":"e827a7223cf119a4ed20a14be44ee6723ac8c0eaa33a520d24abe089cb6212cd"} Sep 11 00:00:01.369004 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:01.368669 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-37692b-29818080-vgfhv" event={"ID":"76dd3efc-c95c-48d4-9910-febb821f1260","Type":"ContainerStarted","Data":"8e49c9122addd006ac6c6e55b470f0f32628c7cd33bc5fa9214192e42564a762"} Sep 11 00:00:01.370211 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:01.370189 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9eee5d-29818080-xp24x" event={"ID":"94f9dae9-20bd-4ab6-b2d9-a103017f8087","Type":"ContainerStarted","Data":"2c8b8dee3440b4b0d26e7a48f28303ff34855971eb5229221ebc511e732dff2c"} Sep 11 00:00:01.370314 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:01.370230 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9eee5d-29818080-xp24x" event={"ID":"94f9dae9-20bd-4ab6-b2d9-a103017f8087","Type":"ContainerStarted","Data":"52f529acfa58647e05ee229a4ce6e1e8b409f588cd7e298b5e1b4109e698403b"} Sep 11 00:00:01.397377 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:01.397330 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9eee5d-29818080-xp24x" podStartSLOduration=1.3973143399999999 podStartE2EDuration="1.39731434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:00:01.395321241 +0000 UTC m=+1416.862795636" watchObservedRunningTime="2026-09-11 00:00:01.39731434 +0000 UTC m=+1416.864788666" Sep 11 00:00:02.376370 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:02.376337 2578 generic.go:356] "Generic (PLEG): container finished" podID="94f9dae9-20bd-4ab6-b2d9-a103017f8087" containerID="2c8b8dee3440b4b0d26e7a48f28303ff34855971eb5229221ebc511e732dff2c" exitCode=0 Sep 11 00:00:02.376860 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:02.376395 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9eee5d-29818080-xp24x" event={"ID":"94f9dae9-20bd-4ab6-b2d9-a103017f8087","Type":"ContainerDied","Data":"2c8b8dee3440b4b0d26e7a48f28303ff34855971eb5229221ebc511e732dff2c"} Sep 11 00:00:02.515198 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:02.515171 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-37692b-29818080-vgfhv" Sep 11 00:00:03.382993 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:03.382967 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-37692b-29818080-vgfhv" Sep 11 00:00:03.382993 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:03.382977 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-37692b-29818080-vgfhv" event={"ID":"76dd3efc-c95c-48d4-9910-febb821f1260","Type":"ContainerDied","Data":"8e49c9122addd006ac6c6e55b470f0f32628c7cd33bc5fa9214192e42564a762"} Sep 11 00:00:03.383430 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:03.383023 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e49c9122addd006ac6c6e55b470f0f32628c7cd33bc5fa9214192e42564a762" Sep 11 00:00:03.517405 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:03.517380 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9eee5d-29818080-xp24x" Sep 11 00:00:04.389004 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:04.388972 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9eee5d-29818080-xp24x" event={"ID":"94f9dae9-20bd-4ab6-b2d9-a103017f8087","Type":"ContainerDied","Data":"52f529acfa58647e05ee229a4ce6e1e8b409f588cd7e298b5e1b4109e698403b"} Sep 11 00:00:04.389384 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:04.389023 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52f529acfa58647e05ee229a4ce6e1e8b409f588cd7e298b5e1b4109e698403b" Sep 11 00:00:04.389384 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:04.388982 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9eee5d-29818080-xp24x" Sep 11 00:00:31.790894 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:31.790808 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794"] Sep 11 00:00:31.791409 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:31.791210 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94f9dae9-20bd-4ab6-b2d9-a103017f8087" containerName="cleanup" Sep 11 00:00:31.791409 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:31.791224 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="94f9dae9-20bd-4ab6-b2d9-a103017f8087" containerName="cleanup" Sep 11 00:00:31.791409 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:31.791256 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76dd3efc-c95c-48d4-9910-febb821f1260" containerName="cleanup" Sep 11 00:00:31.791409 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:31.791263 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="76dd3efc-c95c-48d4-9910-febb821f1260" containerName="cleanup" Sep 11 00:00:31.791409 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:31.791311 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="94f9dae9-20bd-4ab6-b2d9-a103017f8087" containerName="cleanup" Sep 11 00:00:31.791409 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:31.791321 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="76dd3efc-c95c-48d4-9910-febb821f1260" containerName="cleanup" Sep 11 00:00:31.794333 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:31.794311 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794" Sep 11 00:00:31.796626 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:31.796604 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-aef6df\"" Sep 11 00:00:31.796776 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:31.796705 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-aef6df-dockercfg-msm6t\"" Sep 11 00:00:31.805520 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:31.805498 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794"] Sep 11 00:00:31.821953 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:31.821908 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wljc5\" (UniqueName: \"kubernetes.io/projected/83bfa178-a62b-4e01-9f8f-17f1b9ef3398-kube-api-access-wljc5\") pod \"payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794\" (UID: \"83bfa178-a62b-4e01-9f8f-17f1b9ef3398\") " pod="openshift-ingress/payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794" Sep 11 00:00:31.822082 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:31.821976 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83bfa178-a62b-4e01-9f8f-17f1b9ef3398-plugins-config-volume\") pod \"payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794\" (UID: \"83bfa178-a62b-4e01-9f8f-17f1b9ef3398\") " pod="openshift-ingress/payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794" Sep 11 00:00:31.922949 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:31.922897 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wljc5\" (UniqueName: \"kubernetes.io/projected/83bfa178-a62b-4e01-9f8f-17f1b9ef3398-kube-api-access-wljc5\") pod \"payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794\" (UID: \"83bfa178-a62b-4e01-9f8f-17f1b9ef3398\") " pod="openshift-ingress/payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794" Sep 11 00:00:31.923113 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:31.922956 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83bfa178-a62b-4e01-9f8f-17f1b9ef3398-plugins-config-volume\") pod \"payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794\" (UID: \"83bfa178-a62b-4e01-9f8f-17f1b9ef3398\") " pod="openshift-ingress/payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794" Sep 11 00:00:31.923575 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:31.923558 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83bfa178-a62b-4e01-9f8f-17f1b9ef3398-plugins-config-volume\") pod \"payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794\" (UID: \"83bfa178-a62b-4e01-9f8f-17f1b9ef3398\") " pod="openshift-ingress/payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794" Sep 11 00:00:31.931151 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:31.931129 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wljc5\" (UniqueName: \"kubernetes.io/projected/83bfa178-a62b-4e01-9f8f-17f1b9ef3398-kube-api-access-wljc5\") pod \"payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794\" (UID: \"83bfa178-a62b-4e01-9f8f-17f1b9ef3398\") " pod="openshift-ingress/payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794" Sep 11 00:00:32.105617 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:32.105522 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794" Sep 11 00:00:32.378142 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:32.378114 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794"] Sep 11 00:00:32.379947 ip-10-0-139-142 kubenswrapper[2578]: W0911 00:00:32.379910 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83bfa178_a62b_4e01_9f8f_17f1b9ef3398.slice/crio-d6938784d2991be477a0ef5adb4d5bb40e9c3163f79c518dc4c6af3b3cd5a8fb WatchSource:0}: Error finding container d6938784d2991be477a0ef5adb4d5bb40e9c3163f79c518dc4c6af3b3cd5a8fb: Status 404 returned error can't find the container with id d6938784d2991be477a0ef5adb4d5bb40e9c3163f79c518dc4c6af3b3cd5a8fb Sep 11 00:00:32.530221 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:32.530184 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794" event={"ID":"83bfa178-a62b-4e01-9f8f-17f1b9ef3398","Type":"ContainerStarted","Data":"7598da8133789defa25ab30ee8cf4eff40f6adf250981471638db9fe0c00dd8f"} Sep 11 00:00:32.530409 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:32.530241 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794" event={"ID":"83bfa178-a62b-4e01-9f8f-17f1b9ef3398","Type":"ContainerStarted","Data":"d6938784d2991be477a0ef5adb4d5bb40e9c3163f79c518dc4c6af3b3cd5a8fb"} Sep 11 00:00:32.530409 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:32.530261 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794" Sep 11 00:00:32.552511 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:32.552459 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794" podStartSLOduration=1.55244617 podStartE2EDuration="1.55244617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:00:32.550882744 +0000 UTC m=+1448.018357070" watchObservedRunningTime="2026-09-11 00:00:32.55244617 +0000 UTC m=+1448.019920495" Sep 11 00:00:43.538117 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:00:43.538081 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794" Sep 11 00:01:01.169183 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:01.169149 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt"] Sep 11 00:01:01.172535 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:01.172516 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt" Sep 11 00:01:01.178199 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:01.178179 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-5ee885\"" Sep 11 00:01:01.178306 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:01.178269 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-5ee885-dockercfg-j24kz\"" Sep 11 00:01:01.196704 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:01.196679 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt"] Sep 11 00:01:01.210665 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:01.210638 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4"] Sep 11 00:01:01.215488 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:01.215461 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4" Sep 11 00:01:01.226375 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:01.226347 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4"] Sep 11 00:01:01.290921 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:01.290883 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjzc5\" (UniqueName: \"kubernetes.io/projected/fa45765d-86b1-4076-b5ad-2792b4527236-kube-api-access-mjzc5\") pod \"payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt\" (UID: \"fa45765d-86b1-4076-b5ad-2792b4527236\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt" Sep 11 00:01:01.291140 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:01.290960 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa45765d-86b1-4076-b5ad-2792b4527236-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt\" (UID: \"fa45765d-86b1-4076-b5ad-2792b4527236\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt" Sep 11 00:01:01.392451 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:01.392413 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjzc5\" (UniqueName: \"kubernetes.io/projected/fa45765d-86b1-4076-b5ad-2792b4527236-kube-api-access-mjzc5\") pod \"payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt\" (UID: \"fa45765d-86b1-4076-b5ad-2792b4527236\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt" Sep 11 00:01:01.392658 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:01.392464 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa45765d-86b1-4076-b5ad-2792b4527236-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt\" (UID: \"fa45765d-86b1-4076-b5ad-2792b4527236\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt" Sep 11 00:01:01.392658 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:01.392569 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4\" (UID: \"2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a\") " pod="openshift-ingress/payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4" Sep 11 00:01:01.392658 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:01.392614 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss8qh\" (UniqueName: \"kubernetes.io/projected/2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a-kube-api-access-ss8qh\") pod \"payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4\" (UID: \"2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a\") " pod="openshift-ingress/payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4" Sep 11 00:01:01.393225 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:01.393205 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa45765d-86b1-4076-b5ad-2792b4527236-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt\" (UID: \"fa45765d-86b1-4076-b5ad-2792b4527236\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt" Sep 11 00:01:01.402106 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:01.402082 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjzc5\" (UniqueName: \"kubernetes.io/projected/fa45765d-86b1-4076-b5ad-2792b4527236-kube-api-access-mjzc5\") pod \"payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt\" (UID: \"fa45765d-86b1-4076-b5ad-2792b4527236\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt" Sep 11 00:01:01.482553 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:01.482486 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt" Sep 11 00:01:01.493652 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:01.493633 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss8qh\" (UniqueName: \"kubernetes.io/projected/2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a-kube-api-access-ss8qh\") pod \"payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4\" (UID: \"2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a\") " pod="openshift-ingress/payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4" Sep 11 00:01:01.493737 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:01.493722 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4\" (UID: \"2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a\") " pod="openshift-ingress/payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4" Sep 11 00:01:01.494604 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:01.494578 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4\" (UID: \"2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a\") " pod="openshift-ingress/payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4" Sep 11 00:01:01.504709 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:01.504685 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss8qh\" (UniqueName: \"kubernetes.io/projected/2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a-kube-api-access-ss8qh\") pod \"payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4\" (UID: \"2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a\") " pod="openshift-ingress/payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4" Sep 11 00:01:01.529364 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:01.529328 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4" Sep 11 00:01:01.757725 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:01.757700 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt"] Sep 11 00:01:01.760235 ip-10-0-139-142 kubenswrapper[2578]: W0911 00:01:01.760208 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa45765d_86b1_4076_b5ad_2792b4527236.slice/crio-1cb8a489dd68b16551b436bc61d67a8ddfc283d25ed8c586c295cfbeb597788f WatchSource:0}: Error finding container 1cb8a489dd68b16551b436bc61d67a8ddfc283d25ed8c586c295cfbeb597788f: Status 404 returned error can't find the container with id 1cb8a489dd68b16551b436bc61d67a8ddfc283d25ed8c586c295cfbeb597788f Sep 11 00:01:01.829393 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:01.829369 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4"] Sep 11 00:01:01.830647 ip-10-0-139-142 kubenswrapper[2578]: W0911 00:01:01.830621 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dac28b4_6f9c_47f5_a1e7_5606afaa8e8a.slice/crio-962a0ed760195b9226242da5cf2c8e96a96e9d4b86ee373e17bfaca7394e6d59 WatchSource:0}: Error finding container 962a0ed760195b9226242da5cf2c8e96a96e9d4b86ee373e17bfaca7394e6d59: Status 404 returned error can't find the container with id 962a0ed760195b9226242da5cf2c8e96a96e9d4b86ee373e17bfaca7394e6d59 Sep 11 00:01:02.681605 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:02.681568 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4" event={"ID":"2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a","Type":"ContainerStarted","Data":"6fa3add0efdde92e281b5996a040e505f871b1e45df3721bd1c6abd42d005b4f"} Sep 11 00:01:02.682153 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:02.681621 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4" event={"ID":"2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a","Type":"ContainerStarted","Data":"962a0ed760195b9226242da5cf2c8e96a96e9d4b86ee373e17bfaca7394e6d59"} Sep 11 00:01:02.682153 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:02.681757 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4" Sep 11 00:01:02.683464 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:02.683436 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt" event={"ID":"fa45765d-86b1-4076-b5ad-2792b4527236","Type":"ContainerStarted","Data":"710423a4ae86344ac989f3c6f27aed2bd691c9c64c5be7b8c1ec9eb500772b2f"} Sep 11 00:01:02.683605 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:02.683482 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt" event={"ID":"fa45765d-86b1-4076-b5ad-2792b4527236","Type":"ContainerStarted","Data":"1cb8a489dd68b16551b436bc61d67a8ddfc283d25ed8c586c295cfbeb597788f"} Sep 11 00:01:02.683605 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:02.683600 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt" Sep 11 00:01:02.700514 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:02.700442 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4" podStartSLOduration=1.700422263 podStartE2EDuration="1.700422263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:01:02.698654172 +0000 UTC m=+1478.166128496" watchObservedRunningTime="2026-09-11 00:01:02.700422263 +0000 UTC m=+1478.167896603" Sep 11 00:01:02.718767 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:02.718717 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt" podStartSLOduration=1.718703302 podStartE2EDuration="1.718703302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:01:02.717205541 +0000 UTC m=+1478.184679870" watchObservedRunningTime="2026-09-11 00:01:02.718703302 +0000 UTC m=+1478.186177628" Sep 11 00:01:13.690989 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:13.690949 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4" Sep 11 00:01:13.691432 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:13.691078 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt" Sep 11 00:01:25.069005 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:25.068961 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zkr6f_aacd921b-5c15-4f7f-8685-cfceebfe5407/console-operator/2.log" Sep 11 00:01:25.070806 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:25.070776 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zkr6f_aacd921b-5c15-4f7f-8685-cfceebfe5407/console-operator/2.log" Sep 11 00:01:47.874517 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:47.874430 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794"] Sep 11 00:01:47.875121 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:47.874667 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794" podUID="83bfa178-a62b-4e01-9f8f-17f1b9ef3398" containerName="payload-processing" containerID="cri-o://7598da8133789defa25ab30ee8cf4eff40f6adf250981471638db9fe0c00dd8f" gracePeriod=30 Sep 11 00:01:48.124097 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:48.124073 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794" Sep 11 00:01:48.224202 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:48.224168 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83bfa178-a62b-4e01-9f8f-17f1b9ef3398-plugins-config-volume\") pod \"83bfa178-a62b-4e01-9f8f-17f1b9ef3398\" (UID: \"83bfa178-a62b-4e01-9f8f-17f1b9ef3398\") " Sep 11 00:01:48.224397 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:48.224235 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wljc5\" (UniqueName: \"kubernetes.io/projected/83bfa178-a62b-4e01-9f8f-17f1b9ef3398-kube-api-access-wljc5\") pod \"83bfa178-a62b-4e01-9f8f-17f1b9ef3398\" (UID: \"83bfa178-a62b-4e01-9f8f-17f1b9ef3398\") " Sep 11 00:01:48.224679 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:48.224641 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83bfa178-a62b-4e01-9f8f-17f1b9ef3398-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "83bfa178-a62b-4e01-9f8f-17f1b9ef3398" (UID: "83bfa178-a62b-4e01-9f8f-17f1b9ef3398"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:01:48.226756 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:48.226720 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83bfa178-a62b-4e01-9f8f-17f1b9ef3398-kube-api-access-wljc5" (OuterVolumeSpecName: "kube-api-access-wljc5") pod "83bfa178-a62b-4e01-9f8f-17f1b9ef3398" (UID: "83bfa178-a62b-4e01-9f8f-17f1b9ef3398"). InnerVolumeSpecName "kube-api-access-wljc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:01:48.325448 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:48.325403 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83bfa178-a62b-4e01-9f8f-17f1b9ef3398-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:01:48.325448 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:48.325444 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wljc5\" (UniqueName: \"kubernetes.io/projected/83bfa178-a62b-4e01-9f8f-17f1b9ef3398-kube-api-access-wljc5\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:01:48.911568 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:48.911532 2578 generic.go:356] "Generic (PLEG): container finished" podID="83bfa178-a62b-4e01-9f8f-17f1b9ef3398" containerID="7598da8133789defa25ab30ee8cf4eff40f6adf250981471638db9fe0c00dd8f" exitCode=0 Sep 11 00:01:48.912038 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:48.911573 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794" Sep 11 00:01:48.912038 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:48.911615 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794" event={"ID":"83bfa178-a62b-4e01-9f8f-17f1b9ef3398","Type":"ContainerDied","Data":"7598da8133789defa25ab30ee8cf4eff40f6adf250981471638db9fe0c00dd8f"} Sep 11 00:01:48.912038 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:48.911664 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794" event={"ID":"83bfa178-a62b-4e01-9f8f-17f1b9ef3398","Type":"ContainerDied","Data":"d6938784d2991be477a0ef5adb4d5bb40e9c3163f79c518dc4c6af3b3cd5a8fb"} Sep 11 00:01:48.912038 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:48.911694 2578 scope.go:117] "RemoveContainer" containerID="7598da8133789defa25ab30ee8cf4eff40f6adf250981471638db9fe0c00dd8f" Sep 11 00:01:48.920779 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:48.920764 2578 scope.go:117] "RemoveContainer" containerID="7598da8133789defa25ab30ee8cf4eff40f6adf250981471638db9fe0c00dd8f" Sep 11 00:01:48.921049 ip-10-0-139-142 kubenswrapper[2578]: E0911 00:01:48.921031 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7598da8133789defa25ab30ee8cf4eff40f6adf250981471638db9fe0c00dd8f\": container with ID starting with 7598da8133789defa25ab30ee8cf4eff40f6adf250981471638db9fe0c00dd8f not found: ID does not exist" containerID="7598da8133789defa25ab30ee8cf4eff40f6adf250981471638db9fe0c00dd8f" Sep 11 00:01:48.921107 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:48.921056 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7598da8133789defa25ab30ee8cf4eff40f6adf250981471638db9fe0c00dd8f"} err="failed to get container status \"7598da8133789defa25ab30ee8cf4eff40f6adf250981471638db9fe0c00dd8f\": rpc error: code = NotFound desc = could not find container \"7598da8133789defa25ab30ee8cf4eff40f6adf250981471638db9fe0c00dd8f\": container with ID starting with 7598da8133789defa25ab30ee8cf4eff40f6adf250981471638db9fe0c00dd8f not found: ID does not exist" Sep 11 00:01:48.934578 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:48.934554 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794"] Sep 11 00:01:48.939056 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:48.939032 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aef6df-7778c85c5f-8h794"] Sep 11 00:01:49.045398 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:49.045357 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83bfa178-a62b-4e01-9f8f-17f1b9ef3398" path="/var/lib/kubelet/pods/83bfa178-a62b-4e01-9f8f-17f1b9ef3398/volumes" Sep 11 00:01:54.866826 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:54.866789 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5ee885-r4m9k"] Sep 11 00:01:54.867298 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:54.867171 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83bfa178-a62b-4e01-9f8f-17f1b9ef3398" containerName="payload-processing" Sep 11 00:01:54.867298 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:54.867183 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="83bfa178-a62b-4e01-9f8f-17f1b9ef3398" containerName="payload-processing" Sep 11 00:01:54.867298 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:54.867240 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="83bfa178-a62b-4e01-9f8f-17f1b9ef3398" containerName="payload-processing" Sep 11 00:01:54.871622 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:54.871604 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5ee885-r4m9k" Sep 11 00:01:54.873909 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:54.873884 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 00:01:54.874049 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:54.874035 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-22lrj\"" Sep 11 00:01:54.877408 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:54.877385 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5ee885-r4m9k"] Sep 11 00:01:54.882506 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:54.882481 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f30ca653-c4dd-4e1b-bbe6-f898e8f80893-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5ee885-r4m9k\" (UID: \"f30ca653-c4dd-4e1b-bbe6-f898e8f80893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5ee885-r4m9k" Sep 11 00:01:54.983099 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:54.983067 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f30ca653-c4dd-4e1b-bbe6-f898e8f80893-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5ee885-r4m9k\" (UID: \"f30ca653-c4dd-4e1b-bbe6-f898e8f80893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5ee885-r4m9k" Sep 11 00:01:54.983697 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:54.983657 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f30ca653-c4dd-4e1b-bbe6-f898e8f80893-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5ee885-r4m9k\" (UID: \"f30ca653-c4dd-4e1b-bbe6-f898e8f80893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5ee885-r4m9k" Sep 11 00:01:55.182804 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:55.182761 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5ee885-r4m9k" Sep 11 00:01:55.457265 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:55.457241 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5ee885-r4m9k"] Sep 11 00:01:55.459058 ip-10-0-139-142 kubenswrapper[2578]: W0911 00:01:55.459027 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf30ca653_c4dd_4e1b_bbe6_f898e8f80893.slice/crio-7095bca13161204d73dc6df7a7fb631232352158d78b5f7f591a8123a4c5a8c2 WatchSource:0}: Error finding container 7095bca13161204d73dc6df7a7fb631232352158d78b5f7f591a8123a4c5a8c2: Status 404 returned error can't find the container with id 7095bca13161204d73dc6df7a7fb631232352158d78b5f7f591a8123a4c5a8c2 Sep 11 00:01:55.949146 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:55.949108 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5ee885-r4m9k" event={"ID":"f30ca653-c4dd-4e1b-bbe6-f898e8f80893","Type":"ContainerStarted","Data":"2a866841768fca52896465e60934d33a649ac519a6e667542a8cd77c8e573574"} Sep 11 00:01:55.949584 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:55.949162 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5ee885-r4m9k" event={"ID":"f30ca653-c4dd-4e1b-bbe6-f898e8f80893","Type":"ContainerStarted","Data":"7095bca13161204d73dc6df7a7fb631232352158d78b5f7f591a8123a4c5a8c2"} Sep 11 00:01:55.964495 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:55.964449 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5ee885-r4m9k" podStartSLOduration=1.9644354769999999 podStartE2EDuration="1.964435477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:01:55.962363769 +0000 UTC m=+1531.429838096" watchObservedRunningTime="2026-09-11 00:01:55.964435477 +0000 UTC m=+1531.431909802" Sep 11 00:01:56.956564 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:56.956530 2578 generic.go:356] "Generic (PLEG): container finished" podID="f30ca653-c4dd-4e1b-bbe6-f898e8f80893" containerID="2a866841768fca52896465e60934d33a649ac519a6e667542a8cd77c8e573574" exitCode=0 Sep 11 00:01:56.956958 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:56.956590 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5ee885-r4m9k" event={"ID":"f30ca653-c4dd-4e1b-bbe6-f898e8f80893","Type":"ContainerDied","Data":"2a866841768fca52896465e60934d33a649ac519a6e667542a8cd77c8e573574"} Sep 11 00:01:58.102730 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:58.102703 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5ee885-r4m9k" Sep 11 00:01:58.110511 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:58.110490 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f30ca653-c4dd-4e1b-bbe6-f898e8f80893-maas-api-service-ca\") pod \"f30ca653-c4dd-4e1b-bbe6-f898e8f80893\" (UID: \"f30ca653-c4dd-4e1b-bbe6-f898e8f80893\") " Sep 11 00:01:58.110796 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:58.110776 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30ca653-c4dd-4e1b-bbe6-f898e8f80893-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f30ca653-c4dd-4e1b-bbe6-f898e8f80893" (UID: "f30ca653-c4dd-4e1b-bbe6-f898e8f80893"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:01:58.211912 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:58.211837 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f30ca653-c4dd-4e1b-bbe6-f898e8f80893-maas-api-service-ca\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:01:58.967065 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:58.967027 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5ee885-r4m9k" event={"ID":"f30ca653-c4dd-4e1b-bbe6-f898e8f80893","Type":"ContainerDied","Data":"7095bca13161204d73dc6df7a7fb631232352158d78b5f7f591a8123a4c5a8c2"} Sep 11 00:01:58.967065 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:58.967063 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5ee885-r4m9k" Sep 11 00:01:58.967327 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:01:58.967078 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7095bca13161204d73dc6df7a7fb631232352158d78b5f7f591a8123a4c5a8c2" Sep 11 00:02:10.395451 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:10.395414 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4"] Sep 11 00:02:10.396011 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:10.395628 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4" podUID="2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a" containerName="payload-processing" containerID="cri-o://6fa3add0efdde92e281b5996a040e505f871b1e45df3721bd1c6abd42d005b4f" gracePeriod=30 Sep 11 00:02:10.404254 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:10.404224 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt"] Sep 11 00:02:10.404465 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:10.404425 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt" podUID="fa45765d-86b1-4076-b5ad-2792b4527236" containerName="payload-pre-processing" containerID="cri-o://710423a4ae86344ac989f3c6f27aed2bd691c9c64c5be7b8c1ec9eb500772b2f" gracePeriod=30 Sep 11 00:02:10.571021 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:10.570965 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5ee885-r4m9k"] Sep 11 00:02:10.573749 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:10.573655 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5ee885-r4m9k"] Sep 11 00:02:10.651249 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:10.651187 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4" Sep 11 00:02:10.704570 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:10.704542 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt" Sep 11 00:02:10.722487 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:10.722453 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ss8qh\" (UniqueName: \"kubernetes.io/projected/2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a-kube-api-access-ss8qh\") pod \"2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a\" (UID: \"2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a\") " Sep 11 00:02:10.722682 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:10.722533 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a-plugins-config-volume\") pod \"2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a\" (UID: \"2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a\") " Sep 11 00:02:10.723010 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:10.722976 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a" (UID: "2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:02:10.724999 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:10.724969 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a-kube-api-access-ss8qh" (OuterVolumeSpecName: "kube-api-access-ss8qh") pod "2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a" (UID: "2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a"). InnerVolumeSpecName "kube-api-access-ss8qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:02:10.823438 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:10.823398 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjzc5\" (UniqueName: \"kubernetes.io/projected/fa45765d-86b1-4076-b5ad-2792b4527236-kube-api-access-mjzc5\") pod \"fa45765d-86b1-4076-b5ad-2792b4527236\" (UID: \"fa45765d-86b1-4076-b5ad-2792b4527236\") " Sep 11 00:02:10.823617 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:10.823525 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa45765d-86b1-4076-b5ad-2792b4527236-plugins-config-volume\") pod \"fa45765d-86b1-4076-b5ad-2792b4527236\" (UID: \"fa45765d-86b1-4076-b5ad-2792b4527236\") " Sep 11 00:02:10.823696 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:10.823685 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss8qh\" (UniqueName: \"kubernetes.io/projected/2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a-kube-api-access-ss8qh\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:02:10.823731 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:10.823697 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:02:10.823918 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:10.823884 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa45765d-86b1-4076-b5ad-2792b4527236-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fa45765d-86b1-4076-b5ad-2792b4527236" (UID: "fa45765d-86b1-4076-b5ad-2792b4527236"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:02:10.825464 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:10.825444 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa45765d-86b1-4076-b5ad-2792b4527236-kube-api-access-mjzc5" (OuterVolumeSpecName: "kube-api-access-mjzc5") pod "fa45765d-86b1-4076-b5ad-2792b4527236" (UID: "fa45765d-86b1-4076-b5ad-2792b4527236"). InnerVolumeSpecName "kube-api-access-mjzc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:02:10.925078 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:10.924989 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa45765d-86b1-4076-b5ad-2792b4527236-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:02:10.925078 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:10.925019 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjzc5\" (UniqueName: \"kubernetes.io/projected/fa45765d-86b1-4076-b5ad-2792b4527236-kube-api-access-mjzc5\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:02:11.028028 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:11.027996 2578 generic.go:356] "Generic (PLEG): container finished" podID="2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a" containerID="6fa3add0efdde92e281b5996a040e505f871b1e45df3721bd1c6abd42d005b4f" exitCode=0 Sep 11 00:02:11.028212 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:11.028041 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4" Sep 11 00:02:11.028212 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:11.028090 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4" event={"ID":"2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a","Type":"ContainerDied","Data":"6fa3add0efdde92e281b5996a040e505f871b1e45df3721bd1c6abd42d005b4f"} Sep 11 00:02:11.028212 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:11.028146 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4" event={"ID":"2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a","Type":"ContainerDied","Data":"962a0ed760195b9226242da5cf2c8e96a96e9d4b86ee373e17bfaca7394e6d59"} Sep 11 00:02:11.028212 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:11.028180 2578 scope.go:117] "RemoveContainer" containerID="6fa3add0efdde92e281b5996a040e505f871b1e45df3721bd1c6abd42d005b4f" Sep 11 00:02:11.029443 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:11.029419 2578 generic.go:356] "Generic (PLEG): container finished" podID="fa45765d-86b1-4076-b5ad-2792b4527236" containerID="710423a4ae86344ac989f3c6f27aed2bd691c9c64c5be7b8c1ec9eb500772b2f" exitCode=0 Sep 11 00:02:11.029547 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:11.029446 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt" Sep 11 00:02:11.029547 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:11.029456 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt" event={"ID":"fa45765d-86b1-4076-b5ad-2792b4527236","Type":"ContainerDied","Data":"710423a4ae86344ac989f3c6f27aed2bd691c9c64c5be7b8c1ec9eb500772b2f"} Sep 11 00:02:11.029547 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:11.029497 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt" event={"ID":"fa45765d-86b1-4076-b5ad-2792b4527236","Type":"ContainerDied","Data":"1cb8a489dd68b16551b436bc61d67a8ddfc283d25ed8c586c295cfbeb597788f"} Sep 11 00:02:11.038234 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:11.038211 2578 scope.go:117] "RemoveContainer" containerID="6fa3add0efdde92e281b5996a040e505f871b1e45df3721bd1c6abd42d005b4f" Sep 11 00:02:11.038521 ip-10-0-139-142 kubenswrapper[2578]: E0911 00:02:11.038498 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fa3add0efdde92e281b5996a040e505f871b1e45df3721bd1c6abd42d005b4f\": container with ID starting with 6fa3add0efdde92e281b5996a040e505f871b1e45df3721bd1c6abd42d005b4f not found: ID does not exist" containerID="6fa3add0efdde92e281b5996a040e505f871b1e45df3721bd1c6abd42d005b4f" Sep 11 00:02:11.038590 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:11.038533 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fa3add0efdde92e281b5996a040e505f871b1e45df3721bd1c6abd42d005b4f"} err="failed to get container status \"6fa3add0efdde92e281b5996a040e505f871b1e45df3721bd1c6abd42d005b4f\": rpc error: code = NotFound desc = could not find container \"6fa3add0efdde92e281b5996a040e505f871b1e45df3721bd1c6abd42d005b4f\": container with ID starting with 6fa3add0efdde92e281b5996a040e505f871b1e45df3721bd1c6abd42d005b4f not found: ID does not exist" Sep 11 00:02:11.038590 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:11.038557 2578 scope.go:117] "RemoveContainer" containerID="710423a4ae86344ac989f3c6f27aed2bd691c9c64c5be7b8c1ec9eb500772b2f" Sep 11 00:02:11.046896 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:11.046158 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f30ca653-c4dd-4e1b-bbe6-f898e8f80893" path="/var/lib/kubelet/pods/f30ca653-c4dd-4e1b-bbe6-f898e8f80893/volumes" Sep 11 00:02:11.051586 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:11.051471 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4"] Sep 11 00:02:11.051586 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:11.051506 2578 scope.go:117] "RemoveContainer" containerID="710423a4ae86344ac989f3c6f27aed2bd691c9c64c5be7b8c1ec9eb500772b2f" Sep 11 00:02:11.051862 ip-10-0-139-142 kubenswrapper[2578]: E0911 00:02:11.051844 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"710423a4ae86344ac989f3c6f27aed2bd691c9c64c5be7b8c1ec9eb500772b2f\": container with ID starting with 710423a4ae86344ac989f3c6f27aed2bd691c9c64c5be7b8c1ec9eb500772b2f not found: ID does not exist" containerID="710423a4ae86344ac989f3c6f27aed2bd691c9c64c5be7b8c1ec9eb500772b2f" Sep 11 00:02:11.051916 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:11.051869 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"710423a4ae86344ac989f3c6f27aed2bd691c9c64c5be7b8c1ec9eb500772b2f"} err="failed to get container status \"710423a4ae86344ac989f3c6f27aed2bd691c9c64c5be7b8c1ec9eb500772b2f\": rpc error: code = NotFound desc = could not find container \"710423a4ae86344ac989f3c6f27aed2bd691c9c64c5be7b8c1ec9eb500772b2f\": container with ID starting with 710423a4ae86344ac989f3c6f27aed2bd691c9c64c5be7b8c1ec9eb500772b2f not found: ID does not exist" Sep 11 00:02:11.053335 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:11.053314 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5ee885-55dddb445-6gsj4"] Sep 11 00:02:11.062238 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:11.062215 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt"] Sep 11 00:02:11.068710 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:11.068687 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5ee885-7459d8c66-4znnt"] Sep 11 00:02:13.044795 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:13.044759 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a" path="/var/lib/kubelet/pods/2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a/volumes" Sep 11 00:02:13.045254 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:13.045122 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa45765d-86b1-4076-b5ad-2792b4527236" path="/var/lib/kubelet/pods/fa45765d-86b1-4076-b5ad-2792b4527236/volumes" Sep 11 00:02:17.357382 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:17.357341 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9eee5d-qk7m4"] Sep 11 00:02:17.357880 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:17.357860 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f30ca653-c4dd-4e1b-bbe6-f898e8f80893" containerName="revoke-keys" Sep 11 00:02:17.358008 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:17.357885 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f30ca653-c4dd-4e1b-bbe6-f898e8f80893" containerName="revoke-keys" Sep 11 00:02:17.358008 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:17.357908 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa45765d-86b1-4076-b5ad-2792b4527236" containerName="payload-pre-processing" Sep 11 00:02:17.358008 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:17.357918 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa45765d-86b1-4076-b5ad-2792b4527236" containerName="payload-pre-processing" Sep 11 00:02:17.358008 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:17.357969 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a" containerName="payload-processing" Sep 11 00:02:17.358008 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:17.357978 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a" containerName="payload-processing" Sep 11 00:02:17.358255 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:17.358075 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa45765d-86b1-4076-b5ad-2792b4527236" containerName="payload-pre-processing" Sep 11 00:02:17.358255 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:17.358092 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dac28b4-6f9c-47f5-a1e7-5606afaa8e8a" containerName="payload-processing" Sep 11 00:02:17.358255 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:17.358103 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f30ca653-c4dd-4e1b-bbe6-f898e8f80893" containerName="revoke-keys" Sep 11 00:02:17.362604 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:17.362582 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9eee5d-qk7m4" Sep 11 00:02:17.365137 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:17.365112 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-22lrj\"" Sep 11 00:02:17.365437 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:17.365377 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 00:02:17.368797 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:17.368776 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9eee5d-qk7m4"] Sep 11 00:02:17.483650 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:17.483617 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b75386-91b9-4328-baa2-99486a83ffe3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9eee5d-qk7m4\" (UID: \"21b75386-91b9-4328-baa2-99486a83ffe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9eee5d-qk7m4" Sep 11 00:02:17.584451 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:17.584386 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b75386-91b9-4328-baa2-99486a83ffe3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9eee5d-qk7m4\" (UID: \"21b75386-91b9-4328-baa2-99486a83ffe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9eee5d-qk7m4" Sep 11 00:02:17.585106 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:17.585088 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b75386-91b9-4328-baa2-99486a83ffe3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9eee5d-qk7m4\" (UID: \"21b75386-91b9-4328-baa2-99486a83ffe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9eee5d-qk7m4" Sep 11 00:02:17.674296 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:17.674256 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9eee5d-qk7m4" Sep 11 00:02:17.945995 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:17.945864 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9eee5d-qk7m4"] Sep 11 00:02:17.948656 ip-10-0-139-142 kubenswrapper[2578]: W0911 00:02:17.948629 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b75386_91b9_4328_baa2_99486a83ffe3.slice/crio-bcb442ef957dde76584fa3d82736e2b56e4db504514866b1e619500483103acf WatchSource:0}: Error finding container bcb442ef957dde76584fa3d82736e2b56e4db504514866b1e619500483103acf: Status 404 returned error can't find the container with id bcb442ef957dde76584fa3d82736e2b56e4db504514866b1e619500483103acf Sep 11 00:02:18.066027 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:18.065991 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9eee5d-qk7m4" event={"ID":"21b75386-91b9-4328-baa2-99486a83ffe3","Type":"ContainerStarted","Data":"bcb442ef957dde76584fa3d82736e2b56e4db504514866b1e619500483103acf"} Sep 11 00:02:19.072274 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:19.072238 2578 generic.go:356] "Generic (PLEG): container finished" podID="21b75386-91b9-4328-baa2-99486a83ffe3" containerID="6e4d5d7a62031ca3ba2ff0c6cfb59219c215e526550b48e7389cd91fb749909a" exitCode=0 Sep 11 00:02:19.072657 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:19.072327 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9eee5d-qk7m4" event={"ID":"21b75386-91b9-4328-baa2-99486a83ffe3","Type":"ContainerDied","Data":"6e4d5d7a62031ca3ba2ff0c6cfb59219c215e526550b48e7389cd91fb749909a"} Sep 11 00:02:20.210148 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:20.210120 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9eee5d-qk7m4" Sep 11 00:02:20.312437 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:20.312407 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b75386-91b9-4328-baa2-99486a83ffe3-maas-api-service-ca\") pod \"21b75386-91b9-4328-baa2-99486a83ffe3\" (UID: \"21b75386-91b9-4328-baa2-99486a83ffe3\") " Sep 11 00:02:20.312741 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:20.312716 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b75386-91b9-4328-baa2-99486a83ffe3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "21b75386-91b9-4328-baa2-99486a83ffe3" (UID: "21b75386-91b9-4328-baa2-99486a83ffe3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:02:20.413864 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:20.413828 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b75386-91b9-4328-baa2-99486a83ffe3-maas-api-service-ca\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:02:21.083087 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:21.083058 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9eee5d-qk7m4" Sep 11 00:02:21.083260 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:21.083101 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9eee5d-qk7m4" event={"ID":"21b75386-91b9-4328-baa2-99486a83ffe3","Type":"ContainerDied","Data":"bcb442ef957dde76584fa3d82736e2b56e4db504514866b1e619500483103acf"} Sep 11 00:02:21.083260 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:21.083145 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcb442ef957dde76584fa3d82736e2b56e4db504514866b1e619500483103acf" Sep 11 00:02:28.024499 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:28.024461 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9eee5d-29818080-xp24x"] Sep 11 00:02:28.032868 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:28.032810 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9eee5d-29818080-xp24x"] Sep 11 00:02:28.050341 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:28.050301 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7"] Sep 11 00:02:28.050587 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:28.050552 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7" podUID="dd46b037-bcc7-41b5-90c7-a343097ca4d2" containerName="payload-processing" containerID="cri-o://d836cb5114fafde9f00cfa5ee4bc22fc964def850b9086b8e314c5d759d8bcd7" gracePeriod=30 Sep 11 00:02:28.055720 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:28.055694 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58"] Sep 11 00:02:28.055915 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:28.055891 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58" podUID="920003d3-ad43-4a7a-9bff-8cf3a0cd222c" containerName="payload-pre-processing" containerID="cri-o://f9bd76587e89fa6770a815c4a26df53515531cde5911a2ce47990d19c398e636" gracePeriod=30 Sep 11 00:02:28.220227 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:28.220167 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9eee5d-qk7m4"] Sep 11 00:02:28.222980 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:28.222923 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9eee5d-qk7m4"] Sep 11 00:02:28.333649 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:28.333626 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7" Sep 11 00:02:28.340957 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:28.340922 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58" Sep 11 00:02:28.487229 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:28.487195 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd46b037-bcc7-41b5-90c7-a343097ca4d2-plugins-config-volume\") pod \"dd46b037-bcc7-41b5-90c7-a343097ca4d2\" (UID: \"dd46b037-bcc7-41b5-90c7-a343097ca4d2\") " Sep 11 00:02:28.487229 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:28.487239 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvq5j\" (UniqueName: \"kubernetes.io/projected/dd46b037-bcc7-41b5-90c7-a343097ca4d2-kube-api-access-vvq5j\") pod \"dd46b037-bcc7-41b5-90c7-a343097ca4d2\" (UID: \"dd46b037-bcc7-41b5-90c7-a343097ca4d2\") " Sep 11 00:02:28.487471 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:28.487298 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/920003d3-ad43-4a7a-9bff-8cf3a0cd222c-plugins-config-volume\") pod \"920003d3-ad43-4a7a-9bff-8cf3a0cd222c\" (UID: \"920003d3-ad43-4a7a-9bff-8cf3a0cd222c\") " Sep 11 00:02:28.487471 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:28.487358 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrqlt\" (UniqueName: \"kubernetes.io/projected/920003d3-ad43-4a7a-9bff-8cf3a0cd222c-kube-api-access-vrqlt\") pod \"920003d3-ad43-4a7a-9bff-8cf3a0cd222c\" (UID: \"920003d3-ad43-4a7a-9bff-8cf3a0cd222c\") " Sep 11 00:02:28.487673 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:28.487640 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd46b037-bcc7-41b5-90c7-a343097ca4d2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dd46b037-bcc7-41b5-90c7-a343097ca4d2" (UID: "dd46b037-bcc7-41b5-90c7-a343097ca4d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:02:28.487742 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:28.487685 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/920003d3-ad43-4a7a-9bff-8cf3a0cd222c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "920003d3-ad43-4a7a-9bff-8cf3a0cd222c" (UID: "920003d3-ad43-4a7a-9bff-8cf3a0cd222c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:02:28.489441 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:28.489410 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/920003d3-ad43-4a7a-9bff-8cf3a0cd222c-kube-api-access-vrqlt" (OuterVolumeSpecName: "kube-api-access-vrqlt") pod "920003d3-ad43-4a7a-9bff-8cf3a0cd222c" (UID: "920003d3-ad43-4a7a-9bff-8cf3a0cd222c"). InnerVolumeSpecName "kube-api-access-vrqlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:02:28.489599 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:28.489580 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd46b037-bcc7-41b5-90c7-a343097ca4d2-kube-api-access-vvq5j" (OuterVolumeSpecName: "kube-api-access-vvq5j") pod "dd46b037-bcc7-41b5-90c7-a343097ca4d2" (UID: "dd46b037-bcc7-41b5-90c7-a343097ca4d2"). InnerVolumeSpecName "kube-api-access-vvq5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:02:28.588980 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:28.588858 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd46b037-bcc7-41b5-90c7-a343097ca4d2-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:02:28.588980 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:28.588892 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvq5j\" (UniqueName: \"kubernetes.io/projected/dd46b037-bcc7-41b5-90c7-a343097ca4d2-kube-api-access-vvq5j\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:02:28.588980 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:28.588900 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/920003d3-ad43-4a7a-9bff-8cf3a0cd222c-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:02:28.588980 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:28.588910 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrqlt\" (UniqueName: \"kubernetes.io/projected/920003d3-ad43-4a7a-9bff-8cf3a0cd222c-kube-api-access-vrqlt\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:02:29.044474 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:29.044441 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21b75386-91b9-4328-baa2-99486a83ffe3" path="/var/lib/kubelet/pods/21b75386-91b9-4328-baa2-99486a83ffe3/volumes" Sep 11 00:02:29.044842 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:29.044728 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94f9dae9-20bd-4ab6-b2d9-a103017f8087" path="/var/lib/kubelet/pods/94f9dae9-20bd-4ab6-b2d9-a103017f8087/volumes" Sep 11 00:02:29.122452 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:29.122414 2578 generic.go:356] "Generic (PLEG): container finished" podID="920003d3-ad43-4a7a-9bff-8cf3a0cd222c" containerID="f9bd76587e89fa6770a815c4a26df53515531cde5911a2ce47990d19c398e636" exitCode=0 Sep 11 00:02:29.122658 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:29.122462 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58" Sep 11 00:02:29.122658 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:29.122502 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58" event={"ID":"920003d3-ad43-4a7a-9bff-8cf3a0cd222c","Type":"ContainerDied","Data":"f9bd76587e89fa6770a815c4a26df53515531cde5911a2ce47990d19c398e636"} Sep 11 00:02:29.122658 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:29.122554 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58" event={"ID":"920003d3-ad43-4a7a-9bff-8cf3a0cd222c","Type":"ContainerDied","Data":"5dc0ee4796d2f796542eeeb3f6d1494abc11b41346228ad7f0b0cf3fce77dc0b"} Sep 11 00:02:29.122658 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:29.122573 2578 scope.go:117] "RemoveContainer" containerID="f9bd76587e89fa6770a815c4a26df53515531cde5911a2ce47990d19c398e636" Sep 11 00:02:29.124297 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:29.124269 2578 generic.go:356] "Generic (PLEG): container finished" podID="dd46b037-bcc7-41b5-90c7-a343097ca4d2" containerID="d836cb5114fafde9f00cfa5ee4bc22fc964def850b9086b8e314c5d759d8bcd7" exitCode=0 Sep 11 00:02:29.124420 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:29.124312 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7" event={"ID":"dd46b037-bcc7-41b5-90c7-a343097ca4d2","Type":"ContainerDied","Data":"d836cb5114fafde9f00cfa5ee4bc22fc964def850b9086b8e314c5d759d8bcd7"} Sep 11 00:02:29.124420 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:29.124327 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7" Sep 11 00:02:29.124420 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:29.124354 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7" event={"ID":"dd46b037-bcc7-41b5-90c7-a343097ca4d2","Type":"ContainerDied","Data":"eec98cca4c8b11b03ff4e6affc81bd191f1521116f62e35bc752fd12c5a8cd03"} Sep 11 00:02:29.131801 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:29.131780 2578 scope.go:117] "RemoveContainer" containerID="f9bd76587e89fa6770a815c4a26df53515531cde5911a2ce47990d19c398e636" Sep 11 00:02:29.132098 ip-10-0-139-142 kubenswrapper[2578]: E0911 00:02:29.132081 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9bd76587e89fa6770a815c4a26df53515531cde5911a2ce47990d19c398e636\": container with ID starting with f9bd76587e89fa6770a815c4a26df53515531cde5911a2ce47990d19c398e636 not found: ID does not exist" containerID="f9bd76587e89fa6770a815c4a26df53515531cde5911a2ce47990d19c398e636" Sep 11 00:02:29.132152 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:29.132106 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9bd76587e89fa6770a815c4a26df53515531cde5911a2ce47990d19c398e636"} err="failed to get container status \"f9bd76587e89fa6770a815c4a26df53515531cde5911a2ce47990d19c398e636\": rpc error: code = NotFound desc = could not find container \"f9bd76587e89fa6770a815c4a26df53515531cde5911a2ce47990d19c398e636\": container with ID starting with f9bd76587e89fa6770a815c4a26df53515531cde5911a2ce47990d19c398e636 not found: ID does not exist" Sep 11 00:02:29.132152 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:29.132123 2578 scope.go:117] "RemoveContainer" containerID="d836cb5114fafde9f00cfa5ee4bc22fc964def850b9086b8e314c5d759d8bcd7" Sep 11 00:02:29.139762 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:29.139742 2578 scope.go:117] "RemoveContainer" containerID="d836cb5114fafde9f00cfa5ee4bc22fc964def850b9086b8e314c5d759d8bcd7" Sep 11 00:02:29.140100 ip-10-0-139-142 kubenswrapper[2578]: E0911 00:02:29.140076 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d836cb5114fafde9f00cfa5ee4bc22fc964def850b9086b8e314c5d759d8bcd7\": container with ID starting with d836cb5114fafde9f00cfa5ee4bc22fc964def850b9086b8e314c5d759d8bcd7 not found: ID does not exist" containerID="d836cb5114fafde9f00cfa5ee4bc22fc964def850b9086b8e314c5d759d8bcd7" Sep 11 00:02:29.140166 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:29.140103 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d836cb5114fafde9f00cfa5ee4bc22fc964def850b9086b8e314c5d759d8bcd7"} err="failed to get container status \"d836cb5114fafde9f00cfa5ee4bc22fc964def850b9086b8e314c5d759d8bcd7\": rpc error: code = NotFound desc = could not find container \"d836cb5114fafde9f00cfa5ee4bc22fc964def850b9086b8e314c5d759d8bcd7\": container with ID starting with d836cb5114fafde9f00cfa5ee4bc22fc964def850b9086b8e314c5d759d8bcd7 not found: ID does not exist" Sep 11 00:02:29.154741 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:29.154713 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7"] Sep 11 00:02:29.169396 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:29.169369 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9eee5d-7b98bd7947-svcx7"] Sep 11 00:02:29.187431 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:29.187400 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58"] Sep 11 00:02:29.202882 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:29.202855 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9eee5d-54998fc7d6-lrl58"] Sep 11 00:02:31.044438 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:31.044405 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="920003d3-ad43-4a7a-9bff-8cf3a0cd222c" path="/var/lib/kubelet/pods/920003d3-ad43-4a7a-9bff-8cf3a0cd222c/volumes" Sep 11 00:02:31.044806 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:31.044753 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd46b037-bcc7-41b5-90c7-a343097ca4d2" path="/var/lib/kubelet/pods/dd46b037-bcc7-41b5-90c7-a343097ca4d2/volumes" Sep 11 00:02:34.570147 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:34.570110 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr"] Sep 11 00:02:34.570551 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:34.570368 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" podUID="2c820563-d145-4544-b705-374e0506d777" containerName="istio-proxy" containerID="cri-o://e2fac45351d08b7644de89743a7f3f6ed1c20c690233a4e95f834ca43ea67daf" gracePeriod=30 Sep 11 00:02:34.912154 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:34.912117 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dd5958-qzxht"] Sep 11 00:02:34.912474 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:34.912461 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21b75386-91b9-4328-baa2-99486a83ffe3" containerName="revoke-keys" Sep 11 00:02:34.912515 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:34.912475 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="21b75386-91b9-4328-baa2-99486a83ffe3" containerName="revoke-keys" Sep 11 00:02:34.912515 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:34.912491 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="920003d3-ad43-4a7a-9bff-8cf3a0cd222c" containerName="payload-pre-processing" Sep 11 00:02:34.912515 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:34.912497 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="920003d3-ad43-4a7a-9bff-8cf3a0cd222c" containerName="payload-pre-processing" Sep 11 00:02:34.912515 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:34.912506 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd46b037-bcc7-41b5-90c7-a343097ca4d2" containerName="payload-processing" Sep 11 00:02:34.912515 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:34.912511 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd46b037-bcc7-41b5-90c7-a343097ca4d2" containerName="payload-processing" Sep 11 00:02:34.912668 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:34.912564 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="920003d3-ad43-4a7a-9bff-8cf3a0cd222c" containerName="payload-pre-processing" Sep 11 00:02:34.912668 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:34.912573 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd46b037-bcc7-41b5-90c7-a343097ca4d2" containerName="payload-processing" Sep 11 00:02:34.912668 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:34.912579 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="21b75386-91b9-4328-baa2-99486a83ffe3" containerName="revoke-keys" Sep 11 00:02:34.915639 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:34.915619 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dd5958-qzxht" Sep 11 00:02:34.918028 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:34.918003 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 00:02:34.918134 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:34.918006 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-22lrj\"" Sep 11 00:02:34.923291 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:34.923268 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dd5958-qzxht"] Sep 11 00:02:35.050597 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:35.050567 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c788354-702f-4227-a658-f1fd2d9bfb84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-dd5958-qzxht\" (UID: \"0c788354-702f-4227-a658-f1fd2d9bfb84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dd5958-qzxht" Sep 11 00:02:35.151689 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:35.151654 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c788354-702f-4227-a658-f1fd2d9bfb84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-dd5958-qzxht\" (UID: \"0c788354-702f-4227-a658-f1fd2d9bfb84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dd5958-qzxht" Sep 11 00:02:35.152482 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:35.152453 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c788354-702f-4227-a658-f1fd2d9bfb84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-dd5958-qzxht\" (UID: \"0c788354-702f-4227-a658-f1fd2d9bfb84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dd5958-qzxht" Sep 11 00:02:35.227323 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:35.227228 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dd5958-qzxht" Sep 11 00:02:35.496607 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:35.496581 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dd5958-qzxht"] Sep 11 00:02:36.161154 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:36.161113 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dd5958-qzxht" event={"ID":"0c788354-702f-4227-a658-f1fd2d9bfb84","Type":"ContainerStarted","Data":"b28d8abc00285662f8bce7ddbdc47373bf63da26b1fceec1233cbf0791a37604"} Sep 11 00:02:36.161525 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:36.161179 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dd5958-qzxht" event={"ID":"0c788354-702f-4227-a658-f1fd2d9bfb84","Type":"ContainerStarted","Data":"42f84be599f5f554a1af6af386cb25a80bf078ea3b52f530b36ffc61a0b6c617"} Sep 11 00:02:37.167722 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:37.167685 2578 generic.go:356] "Generic (PLEG): container finished" podID="0c788354-702f-4227-a658-f1fd2d9bfb84" containerID="b28d8abc00285662f8bce7ddbdc47373bf63da26b1fceec1233cbf0791a37604" exitCode=0 Sep 11 00:02:37.168152 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:37.167772 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dd5958-qzxht" event={"ID":"0c788354-702f-4227-a658-f1fd2d9bfb84","Type":"ContainerDied","Data":"b28d8abc00285662f8bce7ddbdc47373bf63da26b1fceec1233cbf0791a37604"} Sep 11 00:02:38.306569 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:38.306543 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dd5958-qzxht" Sep 11 00:02:38.379790 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:38.379751 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c788354-702f-4227-a658-f1fd2d9bfb84-maas-api-service-ca\") pod \"0c788354-702f-4227-a658-f1fd2d9bfb84\" (UID: \"0c788354-702f-4227-a658-f1fd2d9bfb84\") " Sep 11 00:02:38.380163 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:38.380137 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c788354-702f-4227-a658-f1fd2d9bfb84-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0c788354-702f-4227-a658-f1fd2d9bfb84" (UID: "0c788354-702f-4227-a658-f1fd2d9bfb84"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:02:38.481388 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:38.481301 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c788354-702f-4227-a658-f1fd2d9bfb84-maas-api-service-ca\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:02:39.180578 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.180534 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dd5958-qzxht" event={"ID":"0c788354-702f-4227-a658-f1fd2d9bfb84","Type":"ContainerDied","Data":"42f84be599f5f554a1af6af386cb25a80bf078ea3b52f530b36ffc61a0b6c617"} Sep 11 00:02:39.180578 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.180571 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dd5958-qzxht" Sep 11 00:02:39.180799 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.180594 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42f84be599f5f554a1af6af386cb25a80bf078ea3b52f530b36ffc61a0b6c617" Sep 11 00:02:39.823164 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.823135 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 11 00:02:39.894178 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.894136 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c820563-d145-4544-b705-374e0506d777-istiod-ca-cert\") pod \"2c820563-d145-4544-b705-374e0506d777\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " Sep 11 00:02:39.894360 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.894189 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-credential-socket\") pod \"2c820563-d145-4544-b705-374e0506d777\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " Sep 11 00:02:39.894360 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.894217 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-workload-socket\") pod \"2c820563-d145-4544-b705-374e0506d777\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " Sep 11 00:02:39.894360 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.894320 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28dtd\" (UniqueName: \"kubernetes.io/projected/2c820563-d145-4544-b705-374e0506d777-kube-api-access-28dtd\") pod \"2c820563-d145-4544-b705-374e0506d777\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " Sep 11 00:02:39.894548 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.894376 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-istio-data\") pod \"2c820563-d145-4544-b705-374e0506d777\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " Sep 11 00:02:39.894548 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.894419 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-workload-certs\") pod \"2c820563-d145-4544-b705-374e0506d777\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " Sep 11 00:02:39.894548 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.894452 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-istio-envoy\") pod \"2c820563-d145-4544-b705-374e0506d777\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " Sep 11 00:02:39.894548 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.894480 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c820563-d145-4544-b705-374e0506d777-istio-token\") pod \"2c820563-d145-4544-b705-374e0506d777\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " Sep 11 00:02:39.894548 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.894520 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2c820563-d145-4544-b705-374e0506d777" (UID: "2c820563-d145-4544-b705-374e0506d777"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:02:39.894548 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.894530 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c820563-d145-4544-b705-374e0506d777-istio-podinfo\") pod \"2c820563-d145-4544-b705-374e0506d777\" (UID: \"2c820563-d145-4544-b705-374e0506d777\") " Sep 11 00:02:39.894548 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.894535 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2c820563-d145-4544-b705-374e0506d777" (UID: "2c820563-d145-4544-b705-374e0506d777"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:02:39.894843 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.894595 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c820563-d145-4544-b705-374e0506d777-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2c820563-d145-4544-b705-374e0506d777" (UID: "2c820563-d145-4544-b705-374e0506d777"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:02:39.894843 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.894679 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-istio-data" (OuterVolumeSpecName: "istio-data") pod "2c820563-d145-4544-b705-374e0506d777" (UID: "2c820563-d145-4544-b705-374e0506d777"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:02:39.895340 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.895283 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2c820563-d145-4544-b705-374e0506d777" (UID: "2c820563-d145-4544-b705-374e0506d777"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:02:39.895617 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.895598 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-workload-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:02:39.895781 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.895764 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-istio-data\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:02:39.895902 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.895889 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c820563-d145-4544-b705-374e0506d777-istiod-ca-cert\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:02:39.896041 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.896025 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-credential-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:02:39.902565 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.902525 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c820563-d145-4544-b705-374e0506d777-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2c820563-d145-4544-b705-374e0506d777" (UID: "2c820563-d145-4544-b705-374e0506d777"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 00:02:39.902565 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.902538 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c820563-d145-4544-b705-374e0506d777-istio-token" (OuterVolumeSpecName: "istio-token") pod "2c820563-d145-4544-b705-374e0506d777" (UID: "2c820563-d145-4544-b705-374e0506d777"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:02:39.902753 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.902685 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2c820563-d145-4544-b705-374e0506d777" (UID: "2c820563-d145-4544-b705-374e0506d777"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:02:39.905279 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.902863 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c820563-d145-4544-b705-374e0506d777-kube-api-access-28dtd" (OuterVolumeSpecName: "kube-api-access-28dtd") pod "2c820563-d145-4544-b705-374e0506d777" (UID: "2c820563-d145-4544-b705-374e0506d777"). InnerVolumeSpecName "kube-api-access-28dtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:02:39.997239 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.997201 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28dtd\" (UniqueName: \"kubernetes.io/projected/2c820563-d145-4544-b705-374e0506d777-kube-api-access-28dtd\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:02:39.997239 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.997234 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-workload-certs\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:02:39.997239 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.997244 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c820563-d145-4544-b705-374e0506d777-istio-envoy\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:02:39.997476 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.997254 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c820563-d145-4544-b705-374e0506d777-istio-token\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:02:39.997476 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:39.997262 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c820563-d145-4544-b705-374e0506d777-istio-podinfo\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:02:40.187195 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:40.187102 2578 generic.go:356] "Generic (PLEG): container finished" podID="2c820563-d145-4544-b705-374e0506d777" containerID="e2fac45351d08b7644de89743a7f3f6ed1c20c690233a4e95f834ca43ea67daf" exitCode=0 Sep 11 00:02:40.187195 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:40.187142 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" event={"ID":"2c820563-d145-4544-b705-374e0506d777","Type":"ContainerDied","Data":"e2fac45351d08b7644de89743a7f3f6ed1c20c690233a4e95f834ca43ea67daf"} Sep 11 00:02:40.187195 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:40.187178 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" Sep 11 00:02:40.187448 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:40.187202 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr" event={"ID":"2c820563-d145-4544-b705-374e0506d777","Type":"ContainerDied","Data":"8a2f39a2cf5f91622061061e339874193a04c0c4f5a917467bb301ad2e94ec26"} Sep 11 00:02:40.187448 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:40.187231 2578 scope.go:117] "RemoveContainer" containerID="e2fac45351d08b7644de89743a7f3f6ed1c20c690233a4e95f834ca43ea67daf" Sep 11 00:02:40.196688 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:40.196658 2578 scope.go:117] "RemoveContainer" containerID="bc266207901f898d579e81ee246e79b2ba3445e653b9d6a2555a8be80ee0d9f2" Sep 11 00:02:40.204840 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:40.204823 2578 scope.go:117] "RemoveContainer" containerID="e2fac45351d08b7644de89743a7f3f6ed1c20c690233a4e95f834ca43ea67daf" Sep 11 00:02:40.205118 ip-10-0-139-142 kubenswrapper[2578]: E0911 00:02:40.205099 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2fac45351d08b7644de89743a7f3f6ed1c20c690233a4e95f834ca43ea67daf\": container with ID starting with e2fac45351d08b7644de89743a7f3f6ed1c20c690233a4e95f834ca43ea67daf not found: ID does not exist" containerID="e2fac45351d08b7644de89743a7f3f6ed1c20c690233a4e95f834ca43ea67daf" Sep 11 00:02:40.205172 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:40.205127 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2fac45351d08b7644de89743a7f3f6ed1c20c690233a4e95f834ca43ea67daf"} err="failed to get container status \"e2fac45351d08b7644de89743a7f3f6ed1c20c690233a4e95f834ca43ea67daf\": rpc error: code = NotFound desc = could not find container \"e2fac45351d08b7644de89743a7f3f6ed1c20c690233a4e95f834ca43ea67daf\": container with ID starting with e2fac45351d08b7644de89743a7f3f6ed1c20c690233a4e95f834ca43ea67daf not found: ID does not exist" Sep 11 00:02:40.205172 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:40.205142 2578 scope.go:117] "RemoveContainer" containerID="bc266207901f898d579e81ee246e79b2ba3445e653b9d6a2555a8be80ee0d9f2" Sep 11 00:02:40.205408 ip-10-0-139-142 kubenswrapper[2578]: E0911 00:02:40.205388 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc266207901f898d579e81ee246e79b2ba3445e653b9d6a2555a8be80ee0d9f2\": container with ID starting with bc266207901f898d579e81ee246e79b2ba3445e653b9d6a2555a8be80ee0d9f2 not found: ID does not exist" containerID="bc266207901f898d579e81ee246e79b2ba3445e653b9d6a2555a8be80ee0d9f2" Sep 11 00:02:40.205473 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:40.205412 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc266207901f898d579e81ee246e79b2ba3445e653b9d6a2555a8be80ee0d9f2"} err="failed to get container status \"bc266207901f898d579e81ee246e79b2ba3445e653b9d6a2555a8be80ee0d9f2\": rpc error: code = NotFound desc = could not find container \"bc266207901f898d579e81ee246e79b2ba3445e653b9d6a2555a8be80ee0d9f2\": container with ID starting with bc266207901f898d579e81ee246e79b2ba3445e653b9d6a2555a8be80ee0d9f2 not found: ID does not exist" Sep 11 00:02:40.210088 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:40.210067 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr"] Sep 11 00:02:40.216651 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:40.216629 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9eee5d-openshift-default-77968dbcfb-4b9zr"] Sep 11 00:02:41.043867 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:41.043832 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c820563-d145-4544-b705-374e0506d777" path="/var/lib/kubelet/pods/2c820563-d145-4544-b705-374e0506d777/volumes" Sep 11 00:02:45.426263 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:45.426229 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn"] Sep 11 00:02:45.426842 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:45.426474 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn" podUID="20688dce-b817-4241-880a-e61c4e240755" containerName="payload-processing" containerID="cri-o://772cd92b085346354f29a0259c8ac13ffbf8653da489ed723446a9e8c54e67a5" gracePeriod=30 Sep 11 00:02:45.433249 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:45.433220 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds"] Sep 11 00:02:45.433668 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:45.433617 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds" podUID="a775eaba-4ecf-40f7-9190-4d7ada5a92ab" containerName="payload-pre-processing" containerID="cri-o://58bd636540a58b330e042ecf798605b7e92781ef113a11b676592bfdeed9d143" gracePeriod=30 Sep 11 00:02:45.604683 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:45.604652 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dd5958-qzxht"] Sep 11 00:02:45.607386 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:45.607344 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dd5958-qzxht"] Sep 11 00:02:45.710465 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:45.710433 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn" Sep 11 00:02:45.713107 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:45.713075 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds" Sep 11 00:02:45.748873 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:45.748150 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a775eaba-4ecf-40f7-9190-4d7ada5a92ab-plugins-config-volume\") pod \"a775eaba-4ecf-40f7-9190-4d7ada5a92ab\" (UID: \"a775eaba-4ecf-40f7-9190-4d7ada5a92ab\") " Sep 11 00:02:45.748873 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:45.748215 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t45rc\" (UniqueName: \"kubernetes.io/projected/20688dce-b817-4241-880a-e61c4e240755-kube-api-access-t45rc\") pod \"20688dce-b817-4241-880a-e61c4e240755\" (UID: \"20688dce-b817-4241-880a-e61c4e240755\") " Sep 11 00:02:45.748873 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:45.748267 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20688dce-b817-4241-880a-e61c4e240755-plugins-config-volume\") pod \"20688dce-b817-4241-880a-e61c4e240755\" (UID: \"20688dce-b817-4241-880a-e61c4e240755\") " Sep 11 00:02:45.748873 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:45.748303 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pz89k\" (UniqueName: \"kubernetes.io/projected/a775eaba-4ecf-40f7-9190-4d7ada5a92ab-kube-api-access-pz89k\") pod \"a775eaba-4ecf-40f7-9190-4d7ada5a92ab\" (UID: \"a775eaba-4ecf-40f7-9190-4d7ada5a92ab\") " Sep 11 00:02:45.748873 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:45.748784 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20688dce-b817-4241-880a-e61c4e240755-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "20688dce-b817-4241-880a-e61c4e240755" (UID: "20688dce-b817-4241-880a-e61c4e240755"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:02:45.749723 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:45.749270 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a775eaba-4ecf-40f7-9190-4d7ada5a92ab-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a775eaba-4ecf-40f7-9190-4d7ada5a92ab" (UID: "a775eaba-4ecf-40f7-9190-4d7ada5a92ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:02:45.751133 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:45.751107 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a775eaba-4ecf-40f7-9190-4d7ada5a92ab-kube-api-access-pz89k" (OuterVolumeSpecName: "kube-api-access-pz89k") pod "a775eaba-4ecf-40f7-9190-4d7ada5a92ab" (UID: "a775eaba-4ecf-40f7-9190-4d7ada5a92ab"). InnerVolumeSpecName "kube-api-access-pz89k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:02:45.751331 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:45.751309 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20688dce-b817-4241-880a-e61c4e240755-kube-api-access-t45rc" (OuterVolumeSpecName: "kube-api-access-t45rc") pod "20688dce-b817-4241-880a-e61c4e240755" (UID: "20688dce-b817-4241-880a-e61c4e240755"). InnerVolumeSpecName "kube-api-access-t45rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:02:45.850011 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:45.849974 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a775eaba-4ecf-40f7-9190-4d7ada5a92ab-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:02:45.850011 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:45.850004 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t45rc\" (UniqueName: \"kubernetes.io/projected/20688dce-b817-4241-880a-e61c4e240755-kube-api-access-t45rc\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:02:45.850011 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:45.850014 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20688dce-b817-4241-880a-e61c4e240755-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:02:45.850253 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:45.850027 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz89k\" (UniqueName: \"kubernetes.io/projected/a775eaba-4ecf-40f7-9190-4d7ada5a92ab-kube-api-access-pz89k\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:02:46.219852 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:46.219819 2578 generic.go:356] "Generic (PLEG): container finished" podID="20688dce-b817-4241-880a-e61c4e240755" containerID="772cd92b085346354f29a0259c8ac13ffbf8653da489ed723446a9e8c54e67a5" exitCode=0 Sep 11 00:02:46.220087 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:46.219866 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn" Sep 11 00:02:46.220087 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:46.219908 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn" event={"ID":"20688dce-b817-4241-880a-e61c4e240755","Type":"ContainerDied","Data":"772cd92b085346354f29a0259c8ac13ffbf8653da489ed723446a9e8c54e67a5"} Sep 11 00:02:46.220087 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:46.219982 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn" event={"ID":"20688dce-b817-4241-880a-e61c4e240755","Type":"ContainerDied","Data":"d8bb1ff2793ee3538daa88309c3478e109f23c96ba5d0ed1c996a0f0ae020160"} Sep 11 00:02:46.220087 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:46.220015 2578 scope.go:117] "RemoveContainer" containerID="772cd92b085346354f29a0259c8ac13ffbf8653da489ed723446a9e8c54e67a5" Sep 11 00:02:46.221187 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:46.221166 2578 generic.go:356] "Generic (PLEG): container finished" podID="a775eaba-4ecf-40f7-9190-4d7ada5a92ab" containerID="58bd636540a58b330e042ecf798605b7e92781ef113a11b676592bfdeed9d143" exitCode=0 Sep 11 00:02:46.221270 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:46.221198 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds" Sep 11 00:02:46.221270 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:46.221204 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds" event={"ID":"a775eaba-4ecf-40f7-9190-4d7ada5a92ab","Type":"ContainerDied","Data":"58bd636540a58b330e042ecf798605b7e92781ef113a11b676592bfdeed9d143"} Sep 11 00:02:46.221270 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:46.221244 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds" event={"ID":"a775eaba-4ecf-40f7-9190-4d7ada5a92ab","Type":"ContainerDied","Data":"d2240e175519bb596b0cad90fd1b3e07ef21eadfaa62bc1ae0a2ab9e72d9ec23"} Sep 11 00:02:46.229768 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:46.229745 2578 scope.go:117] "RemoveContainer" containerID="772cd92b085346354f29a0259c8ac13ffbf8653da489ed723446a9e8c54e67a5" Sep 11 00:02:46.230136 ip-10-0-139-142 kubenswrapper[2578]: E0911 00:02:46.230105 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"772cd92b085346354f29a0259c8ac13ffbf8653da489ed723446a9e8c54e67a5\": container with ID starting with 772cd92b085346354f29a0259c8ac13ffbf8653da489ed723446a9e8c54e67a5 not found: ID does not exist" containerID="772cd92b085346354f29a0259c8ac13ffbf8653da489ed723446a9e8c54e67a5" Sep 11 00:02:46.230206 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:46.230145 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"772cd92b085346354f29a0259c8ac13ffbf8653da489ed723446a9e8c54e67a5"} err="failed to get container status \"772cd92b085346354f29a0259c8ac13ffbf8653da489ed723446a9e8c54e67a5\": rpc error: code = NotFound desc = could not find container \"772cd92b085346354f29a0259c8ac13ffbf8653da489ed723446a9e8c54e67a5\": container with ID starting with 772cd92b085346354f29a0259c8ac13ffbf8653da489ed723446a9e8c54e67a5 not found: ID does not exist" Sep 11 00:02:46.230206 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:46.230168 2578 scope.go:117] "RemoveContainer" containerID="58bd636540a58b330e042ecf798605b7e92781ef113a11b676592bfdeed9d143" Sep 11 00:02:46.238305 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:46.238287 2578 scope.go:117] "RemoveContainer" containerID="58bd636540a58b330e042ecf798605b7e92781ef113a11b676592bfdeed9d143" Sep 11 00:02:46.238553 ip-10-0-139-142 kubenswrapper[2578]: E0911 00:02:46.238536 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58bd636540a58b330e042ecf798605b7e92781ef113a11b676592bfdeed9d143\": container with ID starting with 58bd636540a58b330e042ecf798605b7e92781ef113a11b676592bfdeed9d143 not found: ID does not exist" containerID="58bd636540a58b330e042ecf798605b7e92781ef113a11b676592bfdeed9d143" Sep 11 00:02:46.238603 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:46.238561 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58bd636540a58b330e042ecf798605b7e92781ef113a11b676592bfdeed9d143"} err="failed to get container status \"58bd636540a58b330e042ecf798605b7e92781ef113a11b676592bfdeed9d143\": rpc error: code = NotFound desc = could not find container \"58bd636540a58b330e042ecf798605b7e92781ef113a11b676592bfdeed9d143\": container with ID starting with 58bd636540a58b330e042ecf798605b7e92781ef113a11b676592bfdeed9d143 not found: ID does not exist" Sep 11 00:02:46.244558 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:46.244535 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds"] Sep 11 00:02:46.250441 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:46.250416 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dd5958-687bc659f4-g8qds"] Sep 11 00:02:46.259764 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:46.259739 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn"] Sep 11 00:02:46.263051 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:46.263029 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-dd5958-6c969dddfc-rmfjn"] Sep 11 00:02:47.045368 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:47.045326 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c788354-702f-4227-a658-f1fd2d9bfb84" path="/var/lib/kubelet/pods/0c788354-702f-4227-a658-f1fd2d9bfb84/volumes" Sep 11 00:02:47.045888 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:47.045868 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20688dce-b817-4241-880a-e61c4e240755" path="/var/lib/kubelet/pods/20688dce-b817-4241-880a-e61c4e240755/volumes" Sep 11 00:02:47.046305 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:02:47.046286 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a775eaba-4ecf-40f7-9190-4d7ada5a92ab" path="/var/lib/kubelet/pods/a775eaba-4ecf-40f7-9190-4d7ada5a92ab/volumes" Sep 11 00:03:10.129158 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.129125 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c"] Sep 11 00:03:10.129563 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.129482 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c820563-d145-4544-b705-374e0506d777" containerName="istio-proxy" Sep 11 00:03:10.129563 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.129495 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c820563-d145-4544-b705-374e0506d777" containerName="istio-proxy" Sep 11 00:03:10.129563 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.129503 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c788354-702f-4227-a658-f1fd2d9bfb84" containerName="revoke-keys" Sep 11 00:03:10.129563 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.129509 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c788354-702f-4227-a658-f1fd2d9bfb84" containerName="revoke-keys" Sep 11 00:03:10.129563 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.129522 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a775eaba-4ecf-40f7-9190-4d7ada5a92ab" containerName="payload-pre-processing" Sep 11 00:03:10.129563 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.129528 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a775eaba-4ecf-40f7-9190-4d7ada5a92ab" containerName="payload-pre-processing" Sep 11 00:03:10.129563 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.129538 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20688dce-b817-4241-880a-e61c4e240755" containerName="payload-processing" Sep 11 00:03:10.129563 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.129544 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="20688dce-b817-4241-880a-e61c4e240755" containerName="payload-processing" Sep 11 00:03:10.129563 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.129553 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c820563-d145-4544-b705-374e0506d777" containerName="istio-proxy" Sep 11 00:03:10.129563 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.129558 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c820563-d145-4544-b705-374e0506d777" containerName="istio-proxy" Sep 11 00:03:10.129994 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.129607 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c820563-d145-4544-b705-374e0506d777" containerName="istio-proxy" Sep 11 00:03:10.129994 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.129616 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="20688dce-b817-4241-880a-e61c4e240755" containerName="payload-processing" Sep 11 00:03:10.129994 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.129623 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a775eaba-4ecf-40f7-9190-4d7ada5a92ab" containerName="payload-pre-processing" Sep 11 00:03:10.129994 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.129631 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c788354-702f-4227-a658-f1fd2d9bfb84" containerName="revoke-keys" Sep 11 00:03:10.129994 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.129636 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c820563-d145-4544-b705-374e0506d777" containerName="istio-proxy" Sep 11 00:03:10.133372 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.133347 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.135705 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.135681 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-51d6fb-openshift-default-dockercfg-bm5cn\"" Sep 11 00:03:10.144701 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.144675 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c"] Sep 11 00:03:10.258050 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.258013 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-istio-envoy\") pod \"e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.258050 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.258058 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8d7323-6ca5-439b-b0e0-b060125a130e-istio-podinfo\") pod \"e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.258317 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.258075 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8d7323-6ca5-439b-b0e0-b060125a130e-istiod-ca-cert\") pod \"e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.258317 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.258137 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-workload-socket\") pod \"e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.258317 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.258167 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-credential-socket\") pod \"e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.258317 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.258238 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6n5v\" (UniqueName: \"kubernetes.io/projected/1a8d7323-6ca5-439b-b0e0-b060125a130e-kube-api-access-b6n5v\") pod \"e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.258317 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.258287 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-istio-data\") pod \"e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.258533 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.258347 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-workload-certs\") pod \"e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.258533 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.258371 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8d7323-6ca5-439b-b0e0-b060125a130e-istio-token\") pod \"e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.359129 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.359096 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-credential-socket\") pod \"e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.359298 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.359139 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6n5v\" (UniqueName: \"kubernetes.io/projected/1a8d7323-6ca5-439b-b0e0-b060125a130e-kube-api-access-b6n5v\") pod \"e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.359298 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.359173 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-istio-data\") pod \"e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.359298 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.359201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-workload-certs\") pod \"e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.359298 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.359222 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8d7323-6ca5-439b-b0e0-b060125a130e-istio-token\") pod \"e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.359298 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.359270 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-istio-envoy\") pod \"e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.359559 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.359307 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8d7323-6ca5-439b-b0e0-b060125a130e-istio-podinfo\") pod \"e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.359559 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.359327 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8d7323-6ca5-439b-b0e0-b060125a130e-istiod-ca-cert\") pod \"e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.359559 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.359457 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-workload-socket\") pod \"e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.359559 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.359516 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-credential-socket\") pod \"e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.359779 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.359743 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-workload-certs\") pod \"e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.359837 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.359794 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-workload-socket\") pod \"e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.359948 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.359907 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-istio-data\") pod \"e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.360085 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.360065 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8d7323-6ca5-439b-b0e0-b060125a130e-istiod-ca-cert\") pod \"e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.361641 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.361618 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-istio-envoy\") pod \"e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.361987 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.361964 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8d7323-6ca5-439b-b0e0-b060125a130e-istio-podinfo\") pod \"e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.374491 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.374460 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6n5v\" (UniqueName: \"kubernetes.io/projected/1a8d7323-6ca5-439b-b0e0-b060125a130e-kube-api-access-b6n5v\") pod \"e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.374609 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.374457 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8d7323-6ca5-439b-b0e0-b060125a130e-istio-token\") pod \"e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.446128 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.446041 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:10.724080 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.724048 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c"] Sep 11 00:03:10.725768 ip-10-0-139-142 kubenswrapper[2578]: W0911 00:03:10.725735 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8d7323_6ca5_439b_b0e0_b060125a130e.slice/crio-9fe831199e136c88d6879f919991c66dd886f208d4495d0bd75c18cb45c0510f WatchSource:0}: Error finding container 9fe831199e136c88d6879f919991c66dd886f208d4495d0bd75c18cb45c0510f: Status 404 returned error can't find the container with id 9fe831199e136c88d6879f919991c66dd886f208d4495d0bd75c18cb45c0510f Sep 11 00:03:10.727992 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.727960 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 00:03:10.728067 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.728026 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 00:03:10.728067 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:10.728051 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 00:03:11.355906 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:11.355870 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" event={"ID":"1a8d7323-6ca5-439b-b0e0-b060125a130e","Type":"ContainerStarted","Data":"64b84021baca7bb4459fb57eae6b9e4f52a88c225cf0c006d57ad2d202da8e16"} Sep 11 00:03:11.356307 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:11.355929 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" event={"ID":"1a8d7323-6ca5-439b-b0e0-b060125a130e","Type":"ContainerStarted","Data":"9fe831199e136c88d6879f919991c66dd886f208d4495d0bd75c18cb45c0510f"} Sep 11 00:03:11.375884 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:11.375832 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" podStartSLOduration=1.375816006 podStartE2EDuration="1.375816006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:03:11.375028557 +0000 UTC m=+1606.842502883" watchObservedRunningTime="2026-09-11 00:03:11.375816006 +0000 UTC m=+1606.843290365" Sep 11 00:03:11.447035 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:11.446992 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:11.452266 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:11.452240 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:12.361753 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:12.361721 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:12.362749 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:12.362731 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:15.113500 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:15.113465 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x"] Sep 11 00:03:15.116874 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:15.116855 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x" Sep 11 00:03:15.119301 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:15.119281 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-51d6fb\"" Sep 11 00:03:15.119407 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:15.119338 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-51d6fb-dockercfg-nbjq2\"" Sep 11 00:03:15.130130 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:15.130105 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x"] Sep 11 00:03:15.157997 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:15.157955 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc"] Sep 11 00:03:15.161162 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:15.161146 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc" Sep 11 00:03:15.172449 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:15.172427 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc"] Sep 11 00:03:15.216610 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:15.216571 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45214f17-0662-42f8-bfe9-252df1596e3e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x\" (UID: \"45214f17-0662-42f8-bfe9-252df1596e3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x" Sep 11 00:03:15.216783 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:15.216631 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29jzd\" (UniqueName: \"kubernetes.io/projected/45214f17-0662-42f8-bfe9-252df1596e3e-kube-api-access-29jzd\") pod \"payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x\" (UID: \"45214f17-0662-42f8-bfe9-252df1596e3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x" Sep 11 00:03:15.317896 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:15.317847 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45214f17-0662-42f8-bfe9-252df1596e3e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x\" (UID: \"45214f17-0662-42f8-bfe9-252df1596e3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x" Sep 11 00:03:15.317896 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:15.317906 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89df9278-5224-4616-a88b-a4ecd7e1b55d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc\" (UID: \"89df9278-5224-4616-a88b-a4ecd7e1b55d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc" Sep 11 00:03:15.318137 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:15.318000 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29jzd\" (UniqueName: \"kubernetes.io/projected/45214f17-0662-42f8-bfe9-252df1596e3e-kube-api-access-29jzd\") pod \"payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x\" (UID: \"45214f17-0662-42f8-bfe9-252df1596e3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x" Sep 11 00:03:15.318137 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:15.318076 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6xtr\" (UniqueName: \"kubernetes.io/projected/89df9278-5224-4616-a88b-a4ecd7e1b55d-kube-api-access-g6xtr\") pod \"payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc\" (UID: \"89df9278-5224-4616-a88b-a4ecd7e1b55d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc" Sep 11 00:03:15.318617 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:15.318595 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45214f17-0662-42f8-bfe9-252df1596e3e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x\" (UID: \"45214f17-0662-42f8-bfe9-252df1596e3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x" Sep 11 00:03:15.326579 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:15.326546 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29jzd\" (UniqueName: \"kubernetes.io/projected/45214f17-0662-42f8-bfe9-252df1596e3e-kube-api-access-29jzd\") pod \"payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x\" (UID: \"45214f17-0662-42f8-bfe9-252df1596e3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x" Sep 11 00:03:15.418641 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:15.418606 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6xtr\" (UniqueName: \"kubernetes.io/projected/89df9278-5224-4616-a88b-a4ecd7e1b55d-kube-api-access-g6xtr\") pod \"payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc\" (UID: \"89df9278-5224-4616-a88b-a4ecd7e1b55d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc" Sep 11 00:03:15.418838 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:15.418665 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89df9278-5224-4616-a88b-a4ecd7e1b55d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc\" (UID: \"89df9278-5224-4616-a88b-a4ecd7e1b55d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc" Sep 11 00:03:15.419414 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:15.419394 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89df9278-5224-4616-a88b-a4ecd7e1b55d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc\" (UID: \"89df9278-5224-4616-a88b-a4ecd7e1b55d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc" Sep 11 00:03:15.426804 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:15.426780 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6xtr\" (UniqueName: \"kubernetes.io/projected/89df9278-5224-4616-a88b-a4ecd7e1b55d-kube-api-access-g6xtr\") pod \"payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc\" (UID: \"89df9278-5224-4616-a88b-a4ecd7e1b55d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc" Sep 11 00:03:15.426955 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:15.426901 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x" Sep 11 00:03:15.471327 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:15.471292 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc" Sep 11 00:03:15.698630 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:15.698604 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x"] Sep 11 00:03:15.700561 ip-10-0-139-142 kubenswrapper[2578]: W0911 00:03:15.700532 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45214f17_0662_42f8_bfe9_252df1596e3e.slice/crio-dd0b9e75c438cfb1e1f41f339fb0660b5d8550bb28a01083053ad118ed653d73 WatchSource:0}: Error finding container dd0b9e75c438cfb1e1f41f339fb0660b5d8550bb28a01083053ad118ed653d73: Status 404 returned error can't find the container with id dd0b9e75c438cfb1e1f41f339fb0660b5d8550bb28a01083053ad118ed653d73 Sep 11 00:03:15.757853 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:15.757829 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc"] Sep 11 00:03:15.759737 ip-10-0-139-142 kubenswrapper[2578]: W0911 00:03:15.759713 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89df9278_5224_4616_a88b_a4ecd7e1b55d.slice/crio-7f83c449d2a054af78801e1ba78602a23b63d240d3a7aec7f4bbd71ef48d92e7 WatchSource:0}: Error finding container 7f83c449d2a054af78801e1ba78602a23b63d240d3a7aec7f4bbd71ef48d92e7: Status 404 returned error can't find the container with id 7f83c449d2a054af78801e1ba78602a23b63d240d3a7aec7f4bbd71ef48d92e7 Sep 11 00:03:16.386480 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:16.386441 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc" event={"ID":"89df9278-5224-4616-a88b-a4ecd7e1b55d","Type":"ContainerStarted","Data":"f3f93d5c48561b829fe22407eea077bf3ca7f8c1f39186d00223e63411b3aaeb"} Sep 11 00:03:16.386991 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:16.386505 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc" event={"ID":"89df9278-5224-4616-a88b-a4ecd7e1b55d","Type":"ContainerStarted","Data":"7f83c449d2a054af78801e1ba78602a23b63d240d3a7aec7f4bbd71ef48d92e7"} Sep 11 00:03:16.386991 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:16.386537 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc" Sep 11 00:03:16.388246 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:16.388217 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x" event={"ID":"45214f17-0662-42f8-bfe9-252df1596e3e","Type":"ContainerStarted","Data":"43c475a2ceecad877c6f4f26b546bcbdd0e7dfeacfa6b90e220776b9de61254d"} Sep 11 00:03:16.388385 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:16.388257 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x" event={"ID":"45214f17-0662-42f8-bfe9-252df1596e3e","Type":"ContainerStarted","Data":"dd0b9e75c438cfb1e1f41f339fb0660b5d8550bb28a01083053ad118ed653d73"} Sep 11 00:03:16.388385 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:16.388313 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x" Sep 11 00:03:16.402974 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:16.402905 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc" podStartSLOduration=1.4028891940000001 podStartE2EDuration="1.402889194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:03:16.401203485 +0000 UTC m=+1611.868677826" watchObservedRunningTime="2026-09-11 00:03:16.402889194 +0000 UTC m=+1611.870363518" Sep 11 00:03:16.417560 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:16.417495 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x" podStartSLOduration=1.41747259 podStartE2EDuration="1.41747259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:03:16.415499758 +0000 UTC m=+1611.882974083" watchObservedRunningTime="2026-09-11 00:03:16.41747259 +0000 UTC m=+1611.884946919" Sep 11 00:03:27.396015 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:27.395984 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc" Sep 11 00:03:27.396460 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:27.396056 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x" Sep 11 00:03:28.403121 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:28.403071 2578 patch_prober.go:28] interesting pod/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 00:03:28.403514 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:28.403150 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" podUID="1a8d7323-6ca5-439b-b0e0-b060125a130e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 00:03:29.453733 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:29.453700 2578 generic.go:356] "Generic (PLEG): container finished" podID="1a8d7323-6ca5-439b-b0e0-b060125a130e" containerID="64b84021baca7bb4459fb57eae6b9e4f52a88c225cf0c006d57ad2d202da8e16" exitCode=0 Sep 11 00:03:29.454289 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:29.453756 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" event={"ID":"1a8d7323-6ca5-439b-b0e0-b060125a130e","Type":"ContainerDied","Data":"64b84021baca7bb4459fb57eae6b9e4f52a88c225cf0c006d57ad2d202da8e16"} Sep 11 00:03:29.454289 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:29.454265 2578 scope.go:117] "RemoveContainer" containerID="64b84021baca7bb4459fb57eae6b9e4f52a88c225cf0c006d57ad2d202da8e16" Sep 11 00:03:29.455304 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:29.455279 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 00:03:29.455368 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:29.455353 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 00:03:29.455407 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:29.455382 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 00:03:30.463740 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:30.463698 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" event={"ID":"1a8d7323-6ca5-439b-b0e0-b060125a130e","Type":"ContainerStarted","Data":"b88c8af59d74aa45fab813f486840604c36eb28a8c042f860f75739209e0087c"} Sep 11 00:03:31.446788 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:31.446752 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:31.447168 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:31.447143 2578 patch_prober.go:28] interesting pod/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 00:03:31.447374 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:31.447347 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" podUID="1a8d7323-6ca5-439b-b0e0-b060125a130e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 00:03:32.446911 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:32.446457 2578 patch_prober.go:28] interesting pod/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 00:03:32.446911 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:32.446536 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" podUID="1a8d7323-6ca5-439b-b0e0-b060125a130e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 00:03:33.447049 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:33.447005 2578 patch_prober.go:28] interesting pod/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 00:03:33.448107 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:33.447665 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" podUID="1a8d7323-6ca5-439b-b0e0-b060125a130e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 00:03:34.446542 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:34.446491 2578 patch_prober.go:28] interesting pod/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 00:03:34.446850 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:34.446567 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" podUID="1a8d7323-6ca5-439b-b0e0-b060125a130e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 00:03:35.446791 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:35.446754 2578 patch_prober.go:28] interesting pod/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 00:03:35.447451 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:35.447397 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" podUID="1a8d7323-6ca5-439b-b0e0-b060125a130e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 00:03:36.449983 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:36.449950 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:36.450436 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:36.450420 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:36.451139 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:36.451118 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:03:44.215492 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:44.215455 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd"] Sep 11 00:03:44.219138 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:44.219111 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd" Sep 11 00:03:44.221587 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:44.221568 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c416fa\"" Sep 11 00:03:44.221587 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:44.221580 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c416fa-dockercfg-74zd9\"" Sep 11 00:03:44.230095 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:44.230073 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd"] Sep 11 00:03:44.286376 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:44.286342 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab6b430d-a72b-44e2-83a3-a0d68ba87c75-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd\" (UID: \"ab6b430d-a72b-44e2-83a3-a0d68ba87c75\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd" Sep 11 00:03:44.286543 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:44.286403 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsrpn\" (UniqueName: \"kubernetes.io/projected/ab6b430d-a72b-44e2-83a3-a0d68ba87c75-kube-api-access-bsrpn\") pod \"payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd\" (UID: \"ab6b430d-a72b-44e2-83a3-a0d68ba87c75\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd" Sep 11 00:03:44.387504 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:44.387466 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab6b430d-a72b-44e2-83a3-a0d68ba87c75-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd\" (UID: \"ab6b430d-a72b-44e2-83a3-a0d68ba87c75\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd" Sep 11 00:03:44.387703 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:44.387525 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsrpn\" (UniqueName: \"kubernetes.io/projected/ab6b430d-a72b-44e2-83a3-a0d68ba87c75-kube-api-access-bsrpn\") pod \"payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd\" (UID: \"ab6b430d-a72b-44e2-83a3-a0d68ba87c75\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd" Sep 11 00:03:44.388235 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:44.388216 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab6b430d-a72b-44e2-83a3-a0d68ba87c75-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd\" (UID: \"ab6b430d-a72b-44e2-83a3-a0d68ba87c75\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd" Sep 11 00:03:44.395735 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:44.395714 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsrpn\" (UniqueName: \"kubernetes.io/projected/ab6b430d-a72b-44e2-83a3-a0d68ba87c75-kube-api-access-bsrpn\") pod \"payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd\" (UID: \"ab6b430d-a72b-44e2-83a3-a0d68ba87c75\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd" Sep 11 00:03:44.531067 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:44.530970 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd" Sep 11 00:03:44.800040 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:44.800014 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd"] Sep 11 00:03:44.802089 ip-10-0-139-142 kubenswrapper[2578]: W0911 00:03:44.802058 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab6b430d_a72b_44e2_83a3_a0d68ba87c75.slice/crio-c4766bd9a9c2e6b1e47ba883c83fe6330b857946d200a089cf3ec1a779f46d6d WatchSource:0}: Error finding container c4766bd9a9c2e6b1e47ba883c83fe6330b857946d200a089cf3ec1a779f46d6d: Status 404 returned error can't find the container with id c4766bd9a9c2e6b1e47ba883c83fe6330b857946d200a089cf3ec1a779f46d6d Sep 11 00:03:45.542547 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:45.542505 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd" event={"ID":"ab6b430d-a72b-44e2-83a3-a0d68ba87c75","Type":"ContainerStarted","Data":"6501b53ba353b4643f01da35a8d1ec2d3bc77514a21412b50508ee56e13cdc65"} Sep 11 00:03:45.543047 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:45.542573 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd" event={"ID":"ab6b430d-a72b-44e2-83a3-a0d68ba87c75","Type":"ContainerStarted","Data":"c4766bd9a9c2e6b1e47ba883c83fe6330b857946d200a089cf3ec1a779f46d6d"} Sep 11 00:03:45.543047 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:45.542675 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd" Sep 11 00:03:45.559701 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:45.559628 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd" podStartSLOduration=1.559607833 podStartE2EDuration="1.559607833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:03:45.558669881 +0000 UTC m=+1641.026144209" watchObservedRunningTime="2026-09-11 00:03:45.559607833 +0000 UTC m=+1641.027082161" Sep 11 00:03:56.549658 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:03:56.549629 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd" Sep 11 00:04:12.384394 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:12.384357 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd_ab6b430d-a72b-44e2-83a3-a0d68ba87c75/payload-processing/0.log" Sep 11 00:04:53.012787 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:53.012700 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x_45214f17-0662-42f8-bfe9-252df1596e3e/payload-processing/0.log" Sep 11 00:04:53.265464 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:53.265384 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd_ab6b430d-a72b-44e2-83a3-a0d68ba87c75/payload-processing/0.log" Sep 11 00:04:53.796876 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:53.796842 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf"] Sep 11 00:04:53.800261 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:53.800240 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:53.802723 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:53.802703 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-da536f-openshift-default-dockercfg-f8xdp\"" Sep 11 00:04:53.813261 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:53.813232 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf"] Sep 11 00:04:53.913257 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:53.913224 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-credential-socket\") pod \"e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:53.913257 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:53.913259 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-istio-envoy\") pod \"e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:53.913484 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:53.913278 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f182fcf-3d37-429e-9712-ba7400128656-istio-podinfo\") pod \"e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:53.913484 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:53.913298 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-istio-data\") pod \"e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:53.913484 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:53.913413 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f182fcf-3d37-429e-9712-ba7400128656-istiod-ca-cert\") pod \"e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:53.913484 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:53.913444 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-workload-socket\") pod \"e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:53.913484 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:53.913468 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f182fcf-3d37-429e-9712-ba7400128656-istio-token\") pod \"e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:53.913649 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:53.913500 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-workload-certs\") pod \"e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:53.913649 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:53.913517 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdqfp\" (UniqueName: \"kubernetes.io/projected/1f182fcf-3d37-429e-9712-ba7400128656-kube-api-access-mdqfp\") pod \"e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:54.014748 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:54.014706 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-credential-socket\") pod \"e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:54.014748 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:54.014747 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-istio-envoy\") pod \"e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:54.015300 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:54.014764 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f182fcf-3d37-429e-9712-ba7400128656-istio-podinfo\") pod \"e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:54.015300 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:54.014789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-istio-data\") pod \"e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:54.015300 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:54.014837 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f182fcf-3d37-429e-9712-ba7400128656-istiod-ca-cert\") pod \"e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:54.015300 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:54.014862 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-workload-socket\") pod \"e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:54.015300 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:54.014898 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f182fcf-3d37-429e-9712-ba7400128656-istio-token\") pod \"e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:54.015300 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:54.014931 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-workload-certs\") pod \"e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:54.015300 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:54.015057 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdqfp\" (UniqueName: \"kubernetes.io/projected/1f182fcf-3d37-429e-9712-ba7400128656-kube-api-access-mdqfp\") pod \"e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:54.015300 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:54.015258 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-credential-socket\") pod \"e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:54.015748 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:54.015441 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-istio-data\") pod \"e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:54.015748 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:54.015479 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-workload-certs\") pod \"e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:54.015748 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:54.015575 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-workload-socket\") pod \"e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:54.015881 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:54.015757 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f182fcf-3d37-429e-9712-ba7400128656-istiod-ca-cert\") pod \"e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:54.017328 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:54.017306 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-istio-envoy\") pod \"e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:54.017575 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:54.017557 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f182fcf-3d37-429e-9712-ba7400128656-istio-podinfo\") pod \"e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:54.023052 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:54.023023 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f182fcf-3d37-429e-9712-ba7400128656-istio-token\") pod \"e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:54.023152 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:54.023133 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdqfp\" (UniqueName: \"kubernetes.io/projected/1f182fcf-3d37-429e-9712-ba7400128656-kube-api-access-mdqfp\") pod \"e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:54.114542 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:54.114452 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:54.401522 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:54.401487 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf"] Sep 11 00:04:54.403692 ip-10-0-139-142 kubenswrapper[2578]: W0911 00:04:54.403662 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f182fcf_3d37_429e_9712_ba7400128656.slice/crio-884246d94c8f16b070ae1ba2546e0dcefc7de91bdc70feaff8b1bb2490db1348 WatchSource:0}: Error finding container 884246d94c8f16b070ae1ba2546e0dcefc7de91bdc70feaff8b1bb2490db1348: Status 404 returned error can't find the container with id 884246d94c8f16b070ae1ba2546e0dcefc7de91bdc70feaff8b1bb2490db1348 Sep 11 00:04:54.406135 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:54.406098 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 00:04:54.406203 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:54.406182 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 00:04:54.406242 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:54.406221 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 00:04:54.889413 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:54.889382 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" event={"ID":"1f182fcf-3d37-429e-9712-ba7400128656","Type":"ContainerStarted","Data":"e1a172c9706f71b530836078d327942ea0ba61fc2d774e4d162971979c7232b9"} Sep 11 00:04:54.889589 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:54.889434 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" event={"ID":"1f182fcf-3d37-429e-9712-ba7400128656","Type":"ContainerStarted","Data":"884246d94c8f16b070ae1ba2546e0dcefc7de91bdc70feaff8b1bb2490db1348"} Sep 11 00:04:54.912195 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:54.912142 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" podStartSLOduration=1.912126995 podStartE2EDuration="1.912126995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:04:54.909734243 +0000 UTC m=+1710.377208579" watchObservedRunningTime="2026-09-11 00:04:54.912126995 +0000 UTC m=+1710.379601322" Sep 11 00:04:55.114991 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:55.114959 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:55.120227 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:55.120204 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:55.894584 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:55.894552 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:04:55.895602 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:04:55.895584 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:05:17.005245 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:17.005211 2578 generic.go:356] "Generic (PLEG): container finished" podID="1f182fcf-3d37-429e-9712-ba7400128656" containerID="e1a172c9706f71b530836078d327942ea0ba61fc2d774e4d162971979c7232b9" exitCode=0 Sep 11 00:05:17.005672 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:17.005283 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" event={"ID":"1f182fcf-3d37-429e-9712-ba7400128656","Type":"ContainerDied","Data":"e1a172c9706f71b530836078d327942ea0ba61fc2d774e4d162971979c7232b9"} Sep 11 00:05:17.005718 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:17.005690 2578 scope.go:117] "RemoveContainer" containerID="e1a172c9706f71b530836078d327942ea0ba61fc2d774e4d162971979c7232b9" Sep 11 00:05:17.006749 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:17.006720 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 00:05:17.006855 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:17.006801 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 00:05:17.006855 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:17.006849 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 00:05:18.012766 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:18.012731 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" event={"ID":"1f182fcf-3d37-429e-9712-ba7400128656","Type":"ContainerStarted","Data":"0eb9b30ce47ab567629d5860130feb1dcf33e2f17b570380b96b127891d643d9"} Sep 11 00:05:18.115439 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:18.115403 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:05:18.115615 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:18.115558 2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 00:05:18.115615 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:18.115604 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" podUID="1f182fcf-3d37-429e-9712-ba7400128656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 00:05:19.115160 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:19.115120 2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 00:05:19.115544 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:19.115193 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" podUID="1f182fcf-3d37-429e-9712-ba7400128656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 00:05:20.115263 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:20.115225 2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 00:05:20.115653 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:20.115282 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" podUID="1f182fcf-3d37-429e-9712-ba7400128656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 00:05:21.115274 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:21.115238 2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 00:05:21.115649 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:21.115311 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" podUID="1f182fcf-3d37-429e-9712-ba7400128656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 00:05:22.115914 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:22.115875 2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 00:05:22.116419 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:22.115968 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" podUID="1f182fcf-3d37-429e-9712-ba7400128656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 00:05:22.400253 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:22.400217 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-da536f-x4ltl"] Sep 11 00:05:22.403513 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:22.403494 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-da536f-x4ltl" Sep 11 00:05:22.405908 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:22.405884 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 00:05:22.406021 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:22.405887 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-22lrj\"" Sep 11 00:05:22.410244 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:22.410220 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-da536f-x4ltl"] Sep 11 00:05:22.494032 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:22.493995 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f57988-a328-4099-abd7-f01e197d92d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-da536f-x4ltl\" (UID: \"b7f57988-a328-4099-abd7-f01e197d92d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-da536f-x4ltl" Sep 11 00:05:22.594987 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:22.594911 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f57988-a328-4099-abd7-f01e197d92d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-da536f-x4ltl\" (UID: \"b7f57988-a328-4099-abd7-f01e197d92d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-da536f-x4ltl" Sep 11 00:05:22.595570 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:22.595548 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f57988-a328-4099-abd7-f01e197d92d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-da536f-x4ltl\" (UID: \"b7f57988-a328-4099-abd7-f01e197d92d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-da536f-x4ltl" Sep 11 00:05:22.715554 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:22.715467 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-da536f-x4ltl" Sep 11 00:05:23.000686 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:23.000660 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-da536f-x4ltl"] Sep 11 00:05:23.002262 ip-10-0-139-142 kubenswrapper[2578]: W0911 00:05:23.002236 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f57988_a328_4099_abd7_f01e197d92d5.slice/crio-2d2f6aa3e8b29d63a3bd8b03f8aa8ddab400931526960358ae276289d39a5856 WatchSource:0}: Error finding container 2d2f6aa3e8b29d63a3bd8b03f8aa8ddab400931526960358ae276289d39a5856: Status 404 returned error can't find the container with id 2d2f6aa3e8b29d63a3bd8b03f8aa8ddab400931526960358ae276289d39a5856 Sep 11 00:05:23.057491 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:23.057458 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-da536f-x4ltl" event={"ID":"b7f57988-a328-4099-abd7-f01e197d92d5","Type":"ContainerStarted","Data":"2d2f6aa3e8b29d63a3bd8b03f8aa8ddab400931526960358ae276289d39a5856"} Sep 11 00:05:23.115243 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:23.115205 2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 00:05:23.115406 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:23.115270 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" podUID="1f182fcf-3d37-429e-9712-ba7400128656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 00:05:24.064696 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:24.064599 2578 generic.go:356] "Generic (PLEG): container finished" podID="b7f57988-a328-4099-abd7-f01e197d92d5" containerID="44b6e98c79b1b07105f5c6c72c9a2a19b5f0280b4e7ffc62636ce7a66d77b0b2" exitCode=0 Sep 11 00:05:24.065171 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:24.064694 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-da536f-x4ltl" event={"ID":"b7f57988-a328-4099-abd7-f01e197d92d5","Type":"ContainerDied","Data":"44b6e98c79b1b07105f5c6c72c9a2a19b5f0280b4e7ffc62636ce7a66d77b0b2"} Sep 11 00:05:24.119597 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:24.119569 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:05:24.119857 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:24.119839 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:05:24.120690 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:24.120668 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:05:25.201747 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:25.201720 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-da536f-x4ltl" Sep 11 00:05:25.210343 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:25.210317 2578 scope.go:117] "RemoveContainer" containerID="b84a3a825231264fe31ad99127c3751a3d05f2b9d3bcc4d206fe8bb9ebebfffc" Sep 11 00:05:25.322638 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:25.322550 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f57988-a328-4099-abd7-f01e197d92d5-maas-api-service-ca\") pod \"b7f57988-a328-4099-abd7-f01e197d92d5\" (UID: \"b7f57988-a328-4099-abd7-f01e197d92d5\") " Sep 11 00:05:25.322911 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:25.322887 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7f57988-a328-4099-abd7-f01e197d92d5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b7f57988-a328-4099-abd7-f01e197d92d5" (UID: "b7f57988-a328-4099-abd7-f01e197d92d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:05:25.423266 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:25.423232 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7f57988-a328-4099-abd7-f01e197d92d5-maas-api-service-ca\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:05:26.074475 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:26.074445 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-da536f-x4ltl" Sep 11 00:05:26.074655 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:26.074487 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-da536f-x4ltl" event={"ID":"b7f57988-a328-4099-abd7-f01e197d92d5","Type":"ContainerDied","Data":"2d2f6aa3e8b29d63a3bd8b03f8aa8ddab400931526960358ae276289d39a5856"} Sep 11 00:05:26.074655 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:26.074534 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d2f6aa3e8b29d63a3bd8b03f8aa8ddab400931526960358ae276289d39a5856" Sep 11 00:05:28.690285 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:28.690247 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-da536f-x4ltl"] Sep 11 00:05:28.693102 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:28.693070 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-da536f-x4ltl"] Sep 11 00:05:29.044796 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:29.044712 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7f57988-a328-4099-abd7-f01e197d92d5" path="/var/lib/kubelet/pods/b7f57988-a328-4099-abd7-f01e197d92d5/volumes" Sep 11 00:05:34.986385 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:34.986351 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf"] Sep 11 00:05:34.986833 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:34.986552 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" podUID="1f182fcf-3d37-429e-9712-ba7400128656" containerName="istio-proxy" containerID="cri-o://0eb9b30ce47ab567629d5860130feb1dcf33e2f17b570380b96b127891d643d9" gracePeriod=30 Sep 11 00:05:39.122847 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:39.122806 2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 00:05:39.123248 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:39.122871 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" podUID="1f182fcf-3d37-429e-9712-ba7400128656" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 00:05:40.145099 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.145068 2578 generic.go:356] "Generic (PLEG): container finished" podID="1f182fcf-3d37-429e-9712-ba7400128656" containerID="0eb9b30ce47ab567629d5860130feb1dcf33e2f17b570380b96b127891d643d9" exitCode=0 Sep 11 00:05:40.145467 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.145137 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" event={"ID":"1f182fcf-3d37-429e-9712-ba7400128656","Type":"ContainerDied","Data":"0eb9b30ce47ab567629d5860130feb1dcf33e2f17b570380b96b127891d643d9"} Sep 11 00:05:40.145467 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.145204 2578 scope.go:117] "RemoveContainer" containerID="e1a172c9706f71b530836078d327942ea0ba61fc2d774e4d162971979c7232b9" Sep 11 00:05:40.259635 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.259560 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:05:40.366340 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.366303 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f182fcf-3d37-429e-9712-ba7400128656-istiod-ca-cert\") pod \"1f182fcf-3d37-429e-9712-ba7400128656\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " Sep 11 00:05:40.366340 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.366343 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-istio-envoy\") pod \"1f182fcf-3d37-429e-9712-ba7400128656\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " Sep 11 00:05:40.366593 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.366362 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f182fcf-3d37-429e-9712-ba7400128656-istio-token\") pod \"1f182fcf-3d37-429e-9712-ba7400128656\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " Sep 11 00:05:40.366593 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.366413 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdqfp\" (UniqueName: \"kubernetes.io/projected/1f182fcf-3d37-429e-9712-ba7400128656-kube-api-access-mdqfp\") pod \"1f182fcf-3d37-429e-9712-ba7400128656\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " Sep 11 00:05:40.366593 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.366435 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-istio-data\") pod \"1f182fcf-3d37-429e-9712-ba7400128656\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " Sep 11 00:05:40.366593 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.366476 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-workload-certs\") pod \"1f182fcf-3d37-429e-9712-ba7400128656\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " Sep 11 00:05:40.366593 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.366503 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f182fcf-3d37-429e-9712-ba7400128656-istio-podinfo\") pod \"1f182fcf-3d37-429e-9712-ba7400128656\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " Sep 11 00:05:40.366593 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.366557 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-credential-socket\") pod \"1f182fcf-3d37-429e-9712-ba7400128656\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " Sep 11 00:05:40.366901 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.366605 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-workload-socket\") pod \"1f182fcf-3d37-429e-9712-ba7400128656\" (UID: \"1f182fcf-3d37-429e-9712-ba7400128656\") " Sep 11 00:05:40.366901 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.366766 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1f182fcf-3d37-429e-9712-ba7400128656" (UID: "1f182fcf-3d37-429e-9712-ba7400128656"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:05:40.366901 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.366810 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f182fcf-3d37-429e-9712-ba7400128656-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1f182fcf-3d37-429e-9712-ba7400128656" (UID: "1f182fcf-3d37-429e-9712-ba7400128656"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:05:40.366901 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.366807 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-istio-data" (OuterVolumeSpecName: "istio-data") pod "1f182fcf-3d37-429e-9712-ba7400128656" (UID: "1f182fcf-3d37-429e-9712-ba7400128656"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:05:40.367156 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.367009 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1f182fcf-3d37-429e-9712-ba7400128656" (UID: "1f182fcf-3d37-429e-9712-ba7400128656"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:05:40.367156 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.367028 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1f182fcf-3d37-429e-9712-ba7400128656" (UID: "1f182fcf-3d37-429e-9712-ba7400128656"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:05:40.367156 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.367061 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f182fcf-3d37-429e-9712-ba7400128656-istiod-ca-cert\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:05:40.367156 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.367083 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-istio-data\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:05:40.367156 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.367097 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-workload-certs\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:05:40.368564 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.368538 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1f182fcf-3d37-429e-9712-ba7400128656" (UID: "1f182fcf-3d37-429e-9712-ba7400128656"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:05:40.368792 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.368777 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f182fcf-3d37-429e-9712-ba7400128656-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1f182fcf-3d37-429e-9712-ba7400128656" (UID: "1f182fcf-3d37-429e-9712-ba7400128656"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 00:05:40.369058 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.369038 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f182fcf-3d37-429e-9712-ba7400128656-istio-token" (OuterVolumeSpecName: "istio-token") pod "1f182fcf-3d37-429e-9712-ba7400128656" (UID: "1f182fcf-3d37-429e-9712-ba7400128656"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:05:40.369058 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.369048 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f182fcf-3d37-429e-9712-ba7400128656-kube-api-access-mdqfp" (OuterVolumeSpecName: "kube-api-access-mdqfp") pod "1f182fcf-3d37-429e-9712-ba7400128656" (UID: "1f182fcf-3d37-429e-9712-ba7400128656"). InnerVolumeSpecName "kube-api-access-mdqfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:05:40.467756 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.467719 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f182fcf-3d37-429e-9712-ba7400128656-istio-podinfo\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:05:40.467756 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.467750 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-credential-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:05:40.467756 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.467760 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-workload-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:05:40.468033 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.467769 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f182fcf-3d37-429e-9712-ba7400128656-istio-envoy\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:05:40.468033 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.467778 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f182fcf-3d37-429e-9712-ba7400128656-istio-token\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:05:40.468033 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:40.467786 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdqfp\" (UniqueName: \"kubernetes.io/projected/1f182fcf-3d37-429e-9712-ba7400128656-kube-api-access-mdqfp\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:05:41.151715 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:41.151687 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" Sep 11 00:05:41.152136 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:41.151678 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf" event={"ID":"1f182fcf-3d37-429e-9712-ba7400128656","Type":"ContainerDied","Data":"884246d94c8f16b070ae1ba2546e0dcefc7de91bdc70feaff8b1bb2490db1348"} Sep 11 00:05:41.152136 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:41.151816 2578 scope.go:117] "RemoveContainer" containerID="0eb9b30ce47ab567629d5860130feb1dcf33e2f17b570380b96b127891d643d9" Sep 11 00:05:41.169728 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:41.169703 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf"] Sep 11 00:05:41.173369 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:41.173342 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-da536f-openshift-default-74864b546-rg4tf"] Sep 11 00:05:43.043960 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:43.043906 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f182fcf-3d37-429e-9712-ba7400128656" path="/var/lib/kubelet/pods/1f182fcf-3d37-429e-9712-ba7400128656/volumes" Sep 11 00:05:54.053453 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:54.053417 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x"] Sep 11 00:05:54.062952 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:54.062898 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc"] Sep 11 00:05:54.063580 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:54.063527 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc" podUID="89df9278-5224-4616-a88b-a4ecd7e1b55d" containerName="payload-pre-processing" containerID="cri-o://f3f93d5c48561b829fe22407eea077bf3ca7f8c1f39186d00223e63411b3aaeb" gracePeriod=30 Sep 11 00:05:54.216282 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:54.216247 2578 generic.go:356] "Generic (PLEG): container finished" podID="89df9278-5224-4616-a88b-a4ecd7e1b55d" containerID="f3f93d5c48561b829fe22407eea077bf3ca7f8c1f39186d00223e63411b3aaeb" exitCode=0 Sep 11 00:05:54.216476 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:54.216322 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc" event={"ID":"89df9278-5224-4616-a88b-a4ecd7e1b55d","Type":"ContainerDied","Data":"f3f93d5c48561b829fe22407eea077bf3ca7f8c1f39186d00223e63411b3aaeb"} Sep 11 00:05:54.216563 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:54.216544 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x" podUID="45214f17-0662-42f8-bfe9-252df1596e3e" containerName="payload-processing" containerID="cri-o://43c475a2ceecad877c6f4f26b546bcbdd0e7dfeacfa6b90e220776b9de61254d" gracePeriod=30 Sep 11 00:05:54.339976 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:54.339953 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc" Sep 11 00:05:54.402503 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:54.402469 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89df9278-5224-4616-a88b-a4ecd7e1b55d-plugins-config-volume\") pod \"89df9278-5224-4616-a88b-a4ecd7e1b55d\" (UID: \"89df9278-5224-4616-a88b-a4ecd7e1b55d\") " Sep 11 00:05:54.402822 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:54.402793 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6xtr\" (UniqueName: \"kubernetes.io/projected/89df9278-5224-4616-a88b-a4ecd7e1b55d-kube-api-access-g6xtr\") pod \"89df9278-5224-4616-a88b-a4ecd7e1b55d\" (UID: \"89df9278-5224-4616-a88b-a4ecd7e1b55d\") " Sep 11 00:05:54.403859 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:54.403824 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89df9278-5224-4616-a88b-a4ecd7e1b55d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "89df9278-5224-4616-a88b-a4ecd7e1b55d" (UID: "89df9278-5224-4616-a88b-a4ecd7e1b55d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:05:54.407694 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:54.407660 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89df9278-5224-4616-a88b-a4ecd7e1b55d-kube-api-access-g6xtr" (OuterVolumeSpecName: "kube-api-access-g6xtr") pod "89df9278-5224-4616-a88b-a4ecd7e1b55d" (UID: "89df9278-5224-4616-a88b-a4ecd7e1b55d"). InnerVolumeSpecName "kube-api-access-g6xtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:05:54.467633 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:54.467602 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x" Sep 11 00:05:54.504175 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:54.504144 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29jzd\" (UniqueName: \"kubernetes.io/projected/45214f17-0662-42f8-bfe9-252df1596e3e-kube-api-access-29jzd\") pod \"45214f17-0662-42f8-bfe9-252df1596e3e\" (UID: \"45214f17-0662-42f8-bfe9-252df1596e3e\") " Sep 11 00:05:54.504340 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:54.504267 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45214f17-0662-42f8-bfe9-252df1596e3e-plugins-config-volume\") pod \"45214f17-0662-42f8-bfe9-252df1596e3e\" (UID: \"45214f17-0662-42f8-bfe9-252df1596e3e\") " Sep 11 00:05:54.504600 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:54.504568 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6xtr\" (UniqueName: \"kubernetes.io/projected/89df9278-5224-4616-a88b-a4ecd7e1b55d-kube-api-access-g6xtr\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:05:54.504600 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:54.504599 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89df9278-5224-4616-a88b-a4ecd7e1b55d-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:05:54.504757 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:54.504623 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45214f17-0662-42f8-bfe9-252df1596e3e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "45214f17-0662-42f8-bfe9-252df1596e3e" (UID: "45214f17-0662-42f8-bfe9-252df1596e3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:05:54.506254 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:54.506228 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45214f17-0662-42f8-bfe9-252df1596e3e-kube-api-access-29jzd" (OuterVolumeSpecName: "kube-api-access-29jzd") pod "45214f17-0662-42f8-bfe9-252df1596e3e" (UID: "45214f17-0662-42f8-bfe9-252df1596e3e"). InnerVolumeSpecName "kube-api-access-29jzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:05:54.606090 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:54.606003 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45214f17-0662-42f8-bfe9-252df1596e3e-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:05:54.606090 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:54.606036 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29jzd\" (UniqueName: \"kubernetes.io/projected/45214f17-0662-42f8-bfe9-252df1596e3e-kube-api-access-29jzd\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:05:55.222702 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:55.222664 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc" Sep 11 00:05:55.223218 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:55.222670 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc" event={"ID":"89df9278-5224-4616-a88b-a4ecd7e1b55d","Type":"ContainerDied","Data":"7f83c449d2a054af78801e1ba78602a23b63d240d3a7aec7f4bbd71ef48d92e7"} Sep 11 00:05:55.223218 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:55.222804 2578 scope.go:117] "RemoveContainer" containerID="f3f93d5c48561b829fe22407eea077bf3ca7f8c1f39186d00223e63411b3aaeb" Sep 11 00:05:55.224658 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:55.224633 2578 generic.go:356] "Generic (PLEG): container finished" podID="45214f17-0662-42f8-bfe9-252df1596e3e" containerID="43c475a2ceecad877c6f4f26b546bcbdd0e7dfeacfa6b90e220776b9de61254d" exitCode=0 Sep 11 00:05:55.224744 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:55.224669 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x" Sep 11 00:05:55.224812 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:55.224750 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x" event={"ID":"45214f17-0662-42f8-bfe9-252df1596e3e","Type":"ContainerDied","Data":"43c475a2ceecad877c6f4f26b546bcbdd0e7dfeacfa6b90e220776b9de61254d"} Sep 11 00:05:55.224812 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:55.224795 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x" event={"ID":"45214f17-0662-42f8-bfe9-252df1596e3e","Type":"ContainerDied","Data":"dd0b9e75c438cfb1e1f41f339fb0660b5d8550bb28a01083053ad118ed653d73"} Sep 11 00:05:55.232160 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:55.232141 2578 scope.go:117] "RemoveContainer" containerID="43c475a2ceecad877c6f4f26b546bcbdd0e7dfeacfa6b90e220776b9de61254d" Sep 11 00:05:55.239711 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:55.239695 2578 scope.go:117] "RemoveContainer" containerID="43c475a2ceecad877c6f4f26b546bcbdd0e7dfeacfa6b90e220776b9de61254d" Sep 11 00:05:55.240008 ip-10-0-139-142 kubenswrapper[2578]: E0911 00:05:55.239989 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43c475a2ceecad877c6f4f26b546bcbdd0e7dfeacfa6b90e220776b9de61254d\": container with ID starting with 43c475a2ceecad877c6f4f26b546bcbdd0e7dfeacfa6b90e220776b9de61254d not found: ID does not exist" containerID="43c475a2ceecad877c6f4f26b546bcbdd0e7dfeacfa6b90e220776b9de61254d" Sep 11 00:05:55.240092 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:55.240021 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43c475a2ceecad877c6f4f26b546bcbdd0e7dfeacfa6b90e220776b9de61254d"} err="failed to get container status \"43c475a2ceecad877c6f4f26b546bcbdd0e7dfeacfa6b90e220776b9de61254d\": rpc error: code = NotFound desc = could not find container \"43c475a2ceecad877c6f4f26b546bcbdd0e7dfeacfa6b90e220776b9de61254d\": container with ID starting with 43c475a2ceecad877c6f4f26b546bcbdd0e7dfeacfa6b90e220776b9de61254d not found: ID does not exist" Sep 11 00:05:55.244741 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:55.244718 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x"] Sep 11 00:05:55.249380 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:55.249351 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-51d6fb-67b79f7bfc-qbl4x"] Sep 11 00:05:55.260119 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:55.260093 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc"] Sep 11 00:05:55.264692 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:55.264667 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-51d6fb-685b5c8f78-l9hkc"] Sep 11 00:05:57.044451 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:57.044417 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45214f17-0662-42f8-bfe9-252df1596e3e" path="/var/lib/kubelet/pods/45214f17-0662-42f8-bfe9-252df1596e3e/volumes" Sep 11 00:05:57.044827 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:05:57.044757 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89df9278-5224-4616-a88b-a4ecd7e1b55d" path="/var/lib/kubelet/pods/89df9278-5224-4616-a88b-a4ecd7e1b55d/volumes" Sep 11 00:06:25.095324 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:06:25.095286 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zkr6f_aacd921b-5c15-4f7f-8685-cfceebfe5407/console-operator/2.log" Sep 11 00:06:25.099483 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:06:25.099462 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zkr6f_aacd921b-5c15-4f7f-8685-cfceebfe5407/console-operator/2.log" Sep 11 00:06:25.235039 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:06:25.235011 2578 scope.go:117] "RemoveContainer" containerID="2c8b8dee3440b4b0d26e7a48f28303ff34855971eb5229221ebc511e732dff2c" Sep 11 00:07:13.993392 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:13.993355 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c"] Sep 11 00:07:13.993996 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:13.993632 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" podUID="1a8d7323-6ca5-439b-b0e0-b060125a130e" containerName="istio-proxy" containerID="cri-o://b88c8af59d74aa45fab813f486840604c36eb28a8c042f860f75739209e0087c" gracePeriod=30 Sep 11 00:07:14.409488 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:14.409451 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c416fa-xbx2s"] Sep 11 00:07:14.409808 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:14.409797 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7f57988-a328-4099-abd7-f01e197d92d5" containerName="revoke-keys" Sep 11 00:07:14.409854 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:14.409811 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7f57988-a328-4099-abd7-f01e197d92d5" containerName="revoke-keys" Sep 11 00:07:14.409854 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:14.409822 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f182fcf-3d37-429e-9712-ba7400128656" containerName="istio-proxy" Sep 11 00:07:14.409854 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:14.409827 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f182fcf-3d37-429e-9712-ba7400128656" containerName="istio-proxy" Sep 11 00:07:14.409854 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:14.409837 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89df9278-5224-4616-a88b-a4ecd7e1b55d" containerName="payload-pre-processing" Sep 11 00:07:14.409854 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:14.409843 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="89df9278-5224-4616-a88b-a4ecd7e1b55d" containerName="payload-pre-processing" Sep 11 00:07:14.409854 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:14.409851 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45214f17-0662-42f8-bfe9-252df1596e3e" containerName="payload-processing" Sep 11 00:07:14.409854 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:14.409856 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="45214f17-0662-42f8-bfe9-252df1596e3e" containerName="payload-processing" Sep 11 00:07:14.410123 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:14.409910 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="89df9278-5224-4616-a88b-a4ecd7e1b55d" containerName="payload-pre-processing" Sep 11 00:07:14.410123 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:14.409917 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f182fcf-3d37-429e-9712-ba7400128656" containerName="istio-proxy" Sep 11 00:07:14.410123 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:14.409924 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f182fcf-3d37-429e-9712-ba7400128656" containerName="istio-proxy" Sep 11 00:07:14.410123 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:14.409950 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="45214f17-0662-42f8-bfe9-252df1596e3e" containerName="payload-processing" Sep 11 00:07:14.410123 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:14.409958 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7f57988-a328-4099-abd7-f01e197d92d5" containerName="revoke-keys" Sep 11 00:07:14.412876 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:14.412860 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c416fa-xbx2s" Sep 11 00:07:14.414983 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:14.414961 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/252f9fe8-101b-4f84-a999-6c605fa3d610-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c416fa-xbx2s\" (UID: \"252f9fe8-101b-4f84-a999-6c605fa3d610\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c416fa-xbx2s" Sep 11 00:07:14.415650 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:14.415631 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-22lrj\"" Sep 11 00:07:14.415707 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:14.415631 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 00:07:14.421868 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:14.421843 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c416fa-xbx2s"] Sep 11 00:07:14.515963 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:14.515893 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/252f9fe8-101b-4f84-a999-6c605fa3d610-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c416fa-xbx2s\" (UID: \"252f9fe8-101b-4f84-a999-6c605fa3d610\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c416fa-xbx2s" Sep 11 00:07:14.516592 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:14.516566 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/252f9fe8-101b-4f84-a999-6c605fa3d610-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c416fa-xbx2s\" (UID: \"252f9fe8-101b-4f84-a999-6c605fa3d610\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c416fa-xbx2s" Sep 11 00:07:14.724318 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:14.724225 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c416fa-xbx2s" Sep 11 00:07:14.996695 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:14.996658 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c416fa-xbx2s"] Sep 11 00:07:14.998128 ip-10-0-139-142 kubenswrapper[2578]: W0911 00:07:14.998096 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod252f9fe8_101b_4f84_a999_6c605fa3d610.slice/crio-e4ac82643539ffb45881ef5e2c0cdf9ef9bb9874e6f95c770c2420a2b2927335 WatchSource:0}: Error finding container e4ac82643539ffb45881ef5e2c0cdf9ef9bb9874e6f95c770c2420a2b2927335: Status 404 returned error can't find the container with id e4ac82643539ffb45881ef5e2c0cdf9ef9bb9874e6f95c770c2420a2b2927335 Sep 11 00:07:15.616517 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:15.616427 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c416fa-xbx2s" event={"ID":"252f9fe8-101b-4f84-a999-6c605fa3d610","Type":"ContainerStarted","Data":"6c211b00300a777e4cb5b6ebe2e036d06d152013328013020b7f205d27f5f694"} Sep 11 00:07:15.616517 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:15.616479 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c416fa-xbx2s" event={"ID":"252f9fe8-101b-4f84-a999-6c605fa3d610","Type":"ContainerStarted","Data":"e4ac82643539ffb45881ef5e2c0cdf9ef9bb9874e6f95c770c2420a2b2927335"} Sep 11 00:07:15.633076 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:15.633027 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c416fa-xbx2s" podStartSLOduration=1.633012165 podStartE2EDuration="1.633012165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:07:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:07:15.630490987 +0000 UTC m=+1851.097965311" watchObservedRunningTime="2026-09-11 00:07:15.633012165 +0000 UTC m=+1851.100486489" Sep 11 00:07:16.623580 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:16.623538 2578 generic.go:356] "Generic (PLEG): container finished" podID="252f9fe8-101b-4f84-a999-6c605fa3d610" containerID="6c211b00300a777e4cb5b6ebe2e036d06d152013328013020b7f205d27f5f694" exitCode=0 Sep 11 00:07:16.624006 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:16.623624 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c416fa-xbx2s" event={"ID":"252f9fe8-101b-4f84-a999-6c605fa3d610","Type":"ContainerDied","Data":"6c211b00300a777e4cb5b6ebe2e036d06d152013328013020b7f205d27f5f694"} Sep 11 00:07:17.755689 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:17.755668 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c416fa-xbx2s" Sep 11 00:07:17.840108 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:17.840071 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/252f9fe8-101b-4f84-a999-6c605fa3d610-maas-api-service-ca\") pod \"252f9fe8-101b-4f84-a999-6c605fa3d610\" (UID: \"252f9fe8-101b-4f84-a999-6c605fa3d610\") " Sep 11 00:07:17.840411 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:17.840386 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/252f9fe8-101b-4f84-a999-6c605fa3d610-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "252f9fe8-101b-4f84-a999-6c605fa3d610" (UID: "252f9fe8-101b-4f84-a999-6c605fa3d610"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:07:17.941320 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:17.941236 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/252f9fe8-101b-4f84-a999-6c605fa3d610-maas-api-service-ca\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:07:18.636015 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:18.635981 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c416fa-xbx2s" Sep 11 00:07:18.636221 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:18.636017 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c416fa-xbx2s" event={"ID":"252f9fe8-101b-4f84-a999-6c605fa3d610","Type":"ContainerDied","Data":"e4ac82643539ffb45881ef5e2c0cdf9ef9bb9874e6f95c770c2420a2b2927335"} Sep 11 00:07:18.636221 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:18.636068 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4ac82643539ffb45881ef5e2c0cdf9ef9bb9874e6f95c770c2420a2b2927335" Sep 11 00:07:19.251345 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.251283 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:07:19.355513 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.355476 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-istio-data\") pod \"1a8d7323-6ca5-439b-b0e0-b060125a130e\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " Sep 11 00:07:19.355513 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.355513 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-istio-envoy\") pod \"1a8d7323-6ca5-439b-b0e0-b060125a130e\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " Sep 11 00:07:19.355765 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.355556 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8d7323-6ca5-439b-b0e0-b060125a130e-istiod-ca-cert\") pod \"1a8d7323-6ca5-439b-b0e0-b060125a130e\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " Sep 11 00:07:19.355765 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.355581 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8d7323-6ca5-439b-b0e0-b060125a130e-istio-podinfo\") pod \"1a8d7323-6ca5-439b-b0e0-b060125a130e\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " Sep 11 00:07:19.355765 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.355604 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-workload-socket\") pod \"1a8d7323-6ca5-439b-b0e0-b060125a130e\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " Sep 11 00:07:19.355765 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.355650 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-credential-socket\") pod \"1a8d7323-6ca5-439b-b0e0-b060125a130e\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " Sep 11 00:07:19.355765 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.355694 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8d7323-6ca5-439b-b0e0-b060125a130e-istio-token\") pod \"1a8d7323-6ca5-439b-b0e0-b060125a130e\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " Sep 11 00:07:19.355765 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.355718 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-workload-certs\") pod \"1a8d7323-6ca5-439b-b0e0-b060125a130e\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " Sep 11 00:07:19.356104 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.355777 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6n5v\" (UniqueName: \"kubernetes.io/projected/1a8d7323-6ca5-439b-b0e0-b060125a130e-kube-api-access-b6n5v\") pod \"1a8d7323-6ca5-439b-b0e0-b060125a130e\" (UID: \"1a8d7323-6ca5-439b-b0e0-b060125a130e\") " Sep 11 00:07:19.356104 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.355842 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-istio-data" (OuterVolumeSpecName: "istio-data") pod "1a8d7323-6ca5-439b-b0e0-b060125a130e" (UID: "1a8d7323-6ca5-439b-b0e0-b060125a130e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:07:19.356104 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.356049 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1a8d7323-6ca5-439b-b0e0-b060125a130e" (UID: "1a8d7323-6ca5-439b-b0e0-b060125a130e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:07:19.356104 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.356086 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8d7323-6ca5-439b-b0e0-b060125a130e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1a8d7323-6ca5-439b-b0e0-b060125a130e" (UID: "1a8d7323-6ca5-439b-b0e0-b060125a130e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:07:19.356288 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.356132 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-istio-data\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:07:19.356288 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.356138 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1a8d7323-6ca5-439b-b0e0-b060125a130e" (UID: "1a8d7323-6ca5-439b-b0e0-b060125a130e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:07:19.356288 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.356272 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1a8d7323-6ca5-439b-b0e0-b060125a130e" (UID: "1a8d7323-6ca5-439b-b0e0-b060125a130e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:07:19.357868 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.357841 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1a8d7323-6ca5-439b-b0e0-b060125a130e" (UID: "1a8d7323-6ca5-439b-b0e0-b060125a130e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:07:19.358037 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.358018 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8d7323-6ca5-439b-b0e0-b060125a130e-kube-api-access-b6n5v" (OuterVolumeSpecName: "kube-api-access-b6n5v") pod "1a8d7323-6ca5-439b-b0e0-b060125a130e" (UID: "1a8d7323-6ca5-439b-b0e0-b060125a130e"). InnerVolumeSpecName "kube-api-access-b6n5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:07:19.358303 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.358282 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a8d7323-6ca5-439b-b0e0-b060125a130e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1a8d7323-6ca5-439b-b0e0-b060125a130e" (UID: "1a8d7323-6ca5-439b-b0e0-b060125a130e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 00:07:19.358303 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.358289 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8d7323-6ca5-439b-b0e0-b060125a130e-istio-token" (OuterVolumeSpecName: "istio-token") pod "1a8d7323-6ca5-439b-b0e0-b060125a130e" (UID: "1a8d7323-6ca5-439b-b0e0-b060125a130e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:07:19.457529 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.457493 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8d7323-6ca5-439b-b0e0-b060125a130e-istio-token\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:07:19.457529 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.457521 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-workload-certs\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:07:19.457529 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.457531 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6n5v\" (UniqueName: \"kubernetes.io/projected/1a8d7323-6ca5-439b-b0e0-b060125a130e-kube-api-access-b6n5v\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:07:19.457529 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.457541 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-istio-envoy\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:07:19.457790 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.457550 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8d7323-6ca5-439b-b0e0-b060125a130e-istiod-ca-cert\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:07:19.457790 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.457558 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8d7323-6ca5-439b-b0e0-b060125a130e-istio-podinfo\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:07:19.457790 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.457565 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-workload-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:07:19.457790 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.457573 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8d7323-6ca5-439b-b0e0-b060125a130e-credential-socket\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:07:19.643231 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.643196 2578 generic.go:356] "Generic (PLEG): container finished" podID="1a8d7323-6ca5-439b-b0e0-b060125a130e" containerID="b88c8af59d74aa45fab813f486840604c36eb28a8c042f860f75739209e0087c" exitCode=0 Sep 11 00:07:19.643389 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.643247 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" Sep 11 00:07:19.643389 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.643286 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" event={"ID":"1a8d7323-6ca5-439b-b0e0-b060125a130e","Type":"ContainerDied","Data":"b88c8af59d74aa45fab813f486840604c36eb28a8c042f860f75739209e0087c"} Sep 11 00:07:19.643389 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.643342 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c" event={"ID":"1a8d7323-6ca5-439b-b0e0-b060125a130e","Type":"ContainerDied","Data":"9fe831199e136c88d6879f919991c66dd886f208d4495d0bd75c18cb45c0510f"} Sep 11 00:07:19.643389 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.643367 2578 scope.go:117] "RemoveContainer" containerID="b88c8af59d74aa45fab813f486840604c36eb28a8c042f860f75739209e0087c" Sep 11 00:07:19.653146 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.653127 2578 scope.go:117] "RemoveContainer" containerID="64b84021baca7bb4459fb57eae6b9e4f52a88c225cf0c006d57ad2d202da8e16" Sep 11 00:07:19.661100 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.661082 2578 scope.go:117] "RemoveContainer" containerID="b88c8af59d74aa45fab813f486840604c36eb28a8c042f860f75739209e0087c" Sep 11 00:07:19.661361 ip-10-0-139-142 kubenswrapper[2578]: E0911 00:07:19.661343 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b88c8af59d74aa45fab813f486840604c36eb28a8c042f860f75739209e0087c\": container with ID starting with b88c8af59d74aa45fab813f486840604c36eb28a8c042f860f75739209e0087c not found: ID does not exist" containerID="b88c8af59d74aa45fab813f486840604c36eb28a8c042f860f75739209e0087c" Sep 11 00:07:19.661412 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.661368 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b88c8af59d74aa45fab813f486840604c36eb28a8c042f860f75739209e0087c"} err="failed to get container status \"b88c8af59d74aa45fab813f486840604c36eb28a8c042f860f75739209e0087c\": rpc error: code = NotFound desc = could not find container \"b88c8af59d74aa45fab813f486840604c36eb28a8c042f860f75739209e0087c\": container with ID starting with b88c8af59d74aa45fab813f486840604c36eb28a8c042f860f75739209e0087c not found: ID does not exist" Sep 11 00:07:19.661412 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.661382 2578 scope.go:117] "RemoveContainer" containerID="64b84021baca7bb4459fb57eae6b9e4f52a88c225cf0c006d57ad2d202da8e16" Sep 11 00:07:19.661667 ip-10-0-139-142 kubenswrapper[2578]: E0911 00:07:19.661644 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64b84021baca7bb4459fb57eae6b9e4f52a88c225cf0c006d57ad2d202da8e16\": container with ID starting with 64b84021baca7bb4459fb57eae6b9e4f52a88c225cf0c006d57ad2d202da8e16 not found: ID does not exist" containerID="64b84021baca7bb4459fb57eae6b9e4f52a88c225cf0c006d57ad2d202da8e16" Sep 11 00:07:19.661768 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.661673 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b84021baca7bb4459fb57eae6b9e4f52a88c225cf0c006d57ad2d202da8e16"} err="failed to get container status \"64b84021baca7bb4459fb57eae6b9e4f52a88c225cf0c006d57ad2d202da8e16\": rpc error: code = NotFound desc = could not find container \"64b84021baca7bb4459fb57eae6b9e4f52a88c225cf0c006d57ad2d202da8e16\": container with ID starting with 64b84021baca7bb4459fb57eae6b9e4f52a88c225cf0c006d57ad2d202da8e16 not found: ID does not exist" Sep 11 00:07:19.664506 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.664485 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c"] Sep 11 00:07:19.667785 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:19.667764 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-51d6fb-openshift-default-7c6d696494-n8p2c"] Sep 11 00:07:21.044914 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:21.044880 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a8d7323-6ca5-439b-b0e0-b060125a130e" path="/var/lib/kubelet/pods/1a8d7323-6ca5-439b-b0e0-b060125a130e/volumes" Sep 11 00:07:25.044993 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:25.044965 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd"] Sep 11 00:07:25.045500 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:25.045153 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd" podUID="ab6b430d-a72b-44e2-83a3-a0d68ba87c75" containerName="payload-processing" containerID="cri-o://6501b53ba353b4643f01da35a8d1ec2d3bc77514a21412b50508ee56e13cdc65" gracePeriod=30 Sep 11 00:07:25.253747 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:25.253711 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c416fa-xbx2s"] Sep 11 00:07:25.258568 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:25.258527 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c416fa-xbx2s"] Sep 11 00:07:25.305258 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:25.305202 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd" Sep 11 00:07:25.408896 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:25.408859 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsrpn\" (UniqueName: \"kubernetes.io/projected/ab6b430d-a72b-44e2-83a3-a0d68ba87c75-kube-api-access-bsrpn\") pod \"ab6b430d-a72b-44e2-83a3-a0d68ba87c75\" (UID: \"ab6b430d-a72b-44e2-83a3-a0d68ba87c75\") " Sep 11 00:07:25.409106 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:25.408907 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab6b430d-a72b-44e2-83a3-a0d68ba87c75-plugins-config-volume\") pod \"ab6b430d-a72b-44e2-83a3-a0d68ba87c75\" (UID: \"ab6b430d-a72b-44e2-83a3-a0d68ba87c75\") " Sep 11 00:07:25.409268 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:25.409245 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab6b430d-a72b-44e2-83a3-a0d68ba87c75-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ab6b430d-a72b-44e2-83a3-a0d68ba87c75" (UID: "ab6b430d-a72b-44e2-83a3-a0d68ba87c75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:07:25.411090 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:25.411064 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab6b430d-a72b-44e2-83a3-a0d68ba87c75-kube-api-access-bsrpn" (OuterVolumeSpecName: "kube-api-access-bsrpn") pod "ab6b430d-a72b-44e2-83a3-a0d68ba87c75" (UID: "ab6b430d-a72b-44e2-83a3-a0d68ba87c75"). InnerVolumeSpecName "kube-api-access-bsrpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:07:25.509634 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:25.509590 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsrpn\" (UniqueName: \"kubernetes.io/projected/ab6b430d-a72b-44e2-83a3-a0d68ba87c75-kube-api-access-bsrpn\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:07:25.509634 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:25.509636 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab6b430d-a72b-44e2-83a3-a0d68ba87c75-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:07:25.673641 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:25.673605 2578 generic.go:356] "Generic (PLEG): container finished" podID="ab6b430d-a72b-44e2-83a3-a0d68ba87c75" containerID="6501b53ba353b4643f01da35a8d1ec2d3bc77514a21412b50508ee56e13cdc65" exitCode=0 Sep 11 00:07:25.673789 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:25.673660 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd" Sep 11 00:07:25.673789 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:25.673696 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd" event={"ID":"ab6b430d-a72b-44e2-83a3-a0d68ba87c75","Type":"ContainerDied","Data":"6501b53ba353b4643f01da35a8d1ec2d3bc77514a21412b50508ee56e13cdc65"} Sep 11 00:07:25.673789 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:25.673773 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd" event={"ID":"ab6b430d-a72b-44e2-83a3-a0d68ba87c75","Type":"ContainerDied","Data":"c4766bd9a9c2e6b1e47ba883c83fe6330b857946d200a089cf3ec1a779f46d6d"} Sep 11 00:07:25.673916 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:25.673795 2578 scope.go:117] "RemoveContainer" containerID="6501b53ba353b4643f01da35a8d1ec2d3bc77514a21412b50508ee56e13cdc65" Sep 11 00:07:25.682623 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:25.682606 2578 scope.go:117] "RemoveContainer" containerID="6501b53ba353b4643f01da35a8d1ec2d3bc77514a21412b50508ee56e13cdc65" Sep 11 00:07:25.682870 ip-10-0-139-142 kubenswrapper[2578]: E0911 00:07:25.682851 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6501b53ba353b4643f01da35a8d1ec2d3bc77514a21412b50508ee56e13cdc65\": container with ID starting with 6501b53ba353b4643f01da35a8d1ec2d3bc77514a21412b50508ee56e13cdc65 not found: ID does not exist" containerID="6501b53ba353b4643f01da35a8d1ec2d3bc77514a21412b50508ee56e13cdc65" Sep 11 00:07:25.682959 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:25.682876 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6501b53ba353b4643f01da35a8d1ec2d3bc77514a21412b50508ee56e13cdc65"} err="failed to get container status \"6501b53ba353b4643f01da35a8d1ec2d3bc77514a21412b50508ee56e13cdc65\": rpc error: code = NotFound desc = could not find container \"6501b53ba353b4643f01da35a8d1ec2d3bc77514a21412b50508ee56e13cdc65\": container with ID starting with 6501b53ba353b4643f01da35a8d1ec2d3bc77514a21412b50508ee56e13cdc65 not found: ID does not exist" Sep 11 00:07:25.695043 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:25.695018 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd"] Sep 11 00:07:25.700743 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:25.700719 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c416fa-7f4c4cc6f7-jnssd"] Sep 11 00:07:27.043804 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:27.043770 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="252f9fe8-101b-4f84-a999-6c605fa3d610" path="/var/lib/kubelet/pods/252f9fe8-101b-4f84-a999-6c605fa3d610/volumes" Sep 11 00:07:27.044188 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:27.044153 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab6b430d-a72b-44e2-83a3-a0d68ba87c75" path="/var/lib/kubelet/pods/ab6b430d-a72b-44e2-83a3-a0d68ba87c75/volumes" Sep 11 00:07:32.069850 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:32.069806 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-37692b-rjbfs"] Sep 11 00:07:32.070385 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:32.070368 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="252f9fe8-101b-4f84-a999-6c605fa3d610" containerName="revoke-keys" Sep 11 00:07:32.070458 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:32.070391 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="252f9fe8-101b-4f84-a999-6c605fa3d610" containerName="revoke-keys" Sep 11 00:07:32.070458 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:32.070419 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f182fcf-3d37-429e-9712-ba7400128656" containerName="istio-proxy" Sep 11 00:07:32.070458 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:32.070430 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f182fcf-3d37-429e-9712-ba7400128656" containerName="istio-proxy" Sep 11 00:07:32.070458 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:32.070447 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a8d7323-6ca5-439b-b0e0-b060125a130e" containerName="istio-proxy" Sep 11 00:07:32.070458 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:32.070455 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a8d7323-6ca5-439b-b0e0-b060125a130e" containerName="istio-proxy" Sep 11 00:07:32.070700 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:32.070468 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a8d7323-6ca5-439b-b0e0-b060125a130e" containerName="istio-proxy" Sep 11 00:07:32.070700 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:32.070476 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a8d7323-6ca5-439b-b0e0-b060125a130e" containerName="istio-proxy" Sep 11 00:07:32.070700 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:32.070492 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab6b430d-a72b-44e2-83a3-a0d68ba87c75" containerName="payload-processing" Sep 11 00:07:32.070700 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:32.070501 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab6b430d-a72b-44e2-83a3-a0d68ba87c75" containerName="payload-processing" Sep 11 00:07:32.070700 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:32.070583 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a8d7323-6ca5-439b-b0e0-b060125a130e" containerName="istio-proxy" Sep 11 00:07:32.070700 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:32.070595 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab6b430d-a72b-44e2-83a3-a0d68ba87c75" containerName="payload-processing" Sep 11 00:07:32.070700 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:32.070610 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="252f9fe8-101b-4f84-a999-6c605fa3d610" containerName="revoke-keys" Sep 11 00:07:32.073729 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:32.073708 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-37692b-rjbfs" Sep 11 00:07:32.075995 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:32.075969 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-22lrj\"" Sep 11 00:07:32.076117 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:32.075972 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 00:07:32.081146 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:32.080779 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-37692b-rjbfs"] Sep 11 00:07:32.164820 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:32.164785 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbed1b7-c55d-458e-825d-15929e2c5efa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-37692b-rjbfs\" (UID: \"bfbed1b7-c55d-458e-825d-15929e2c5efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-37692b-rjbfs" Sep 11 00:07:32.265401 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:32.265368 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbed1b7-c55d-458e-825d-15929e2c5efa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-37692b-rjbfs\" (UID: \"bfbed1b7-c55d-458e-825d-15929e2c5efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-37692b-rjbfs" Sep 11 00:07:32.266065 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:32.266031 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbed1b7-c55d-458e-825d-15929e2c5efa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-37692b-rjbfs\" (UID: \"bfbed1b7-c55d-458e-825d-15929e2c5efa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-37692b-rjbfs" Sep 11 00:07:32.384913 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:32.384873 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-37692b-rjbfs" Sep 11 00:07:32.658998 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:32.658971 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-37692b-rjbfs"] Sep 11 00:07:32.661141 ip-10-0-139-142 kubenswrapper[2578]: W0911 00:07:32.661117 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfbed1b7_c55d_458e_825d_15929e2c5efa.slice/crio-dacfdfa64b64e3e26c373846fbf0616d05773ea7d6aa2ed31e9c87c12b280c0a WatchSource:0}: Error finding container dacfdfa64b64e3e26c373846fbf0616d05773ea7d6aa2ed31e9c87c12b280c0a: Status 404 returned error can't find the container with id dacfdfa64b64e3e26c373846fbf0616d05773ea7d6aa2ed31e9c87c12b280c0a Sep 11 00:07:32.708800 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:32.708769 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-37692b-rjbfs" event={"ID":"bfbed1b7-c55d-458e-825d-15929e2c5efa","Type":"ContainerStarted","Data":"dacfdfa64b64e3e26c373846fbf0616d05773ea7d6aa2ed31e9c87c12b280c0a"} Sep 11 00:07:33.714693 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:33.714654 2578 generic.go:356] "Generic (PLEG): container finished" podID="bfbed1b7-c55d-458e-825d-15929e2c5efa" containerID="4b500013dff32e4adefeaf914f2fd3d0eff9de39dc4a9e4df5465bae2fbfbd5a" exitCode=0 Sep 11 00:07:33.715142 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:33.714741 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-37692b-rjbfs" event={"ID":"bfbed1b7-c55d-458e-825d-15929e2c5efa","Type":"ContainerDied","Data":"4b500013dff32e4adefeaf914f2fd3d0eff9de39dc4a9e4df5465bae2fbfbd5a"} Sep 11 00:07:34.842197 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:34.842163 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-37692b-rjbfs" Sep 11 00:07:34.890515 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:34.890479 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbed1b7-c55d-458e-825d-15929e2c5efa-maas-api-service-ca\") pod \"bfbed1b7-c55d-458e-825d-15929e2c5efa\" (UID: \"bfbed1b7-c55d-458e-825d-15929e2c5efa\") " Sep 11 00:07:34.890961 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:34.890917 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfbed1b7-c55d-458e-825d-15929e2c5efa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bfbed1b7-c55d-458e-825d-15929e2c5efa" (UID: "bfbed1b7-c55d-458e-825d-15929e2c5efa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:07:34.991406 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:34.991321 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbed1b7-c55d-458e-825d-15929e2c5efa-maas-api-service-ca\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:07:35.725201 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:35.725174 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-37692b-rjbfs" Sep 11 00:07:35.725377 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:35.725179 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-37692b-rjbfs" event={"ID":"bfbed1b7-c55d-458e-825d-15929e2c5efa","Type":"ContainerDied","Data":"dacfdfa64b64e3e26c373846fbf0616d05773ea7d6aa2ed31e9c87c12b280c0a"} Sep 11 00:07:35.725377 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:35.725299 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dacfdfa64b64e3e26c373846fbf0616d05773ea7d6aa2ed31e9c87c12b280c0a" Sep 11 00:07:42.605144 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:42.605094 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-37692b-29818080-vgfhv"] Sep 11 00:07:42.610399 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:42.610373 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-37692b-29818080-vgfhv"] Sep 11 00:07:42.642743 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:42.642711 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k"] Sep 11 00:07:42.643003 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:42.642958 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k" podUID="3e0c579f-83f1-4f7a-b52f-a71d18df718a" containerName="payload-processing" containerID="cri-o://34b1cf77906f5d84d8b720095071470ec89204586d77995b8920fb28c3284231" gracePeriod=30 Sep 11 00:07:42.653321 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:42.653294 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd"] Sep 11 00:07:42.653698 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:42.653648 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd" podUID="8d53a069-c3ba-4963-b577-fd3035240d4e" containerName="payload-pre-processing" containerID="cri-o://b3b1001d7dd45ed272552548abb05786d446c6dafbdea89be3bb7d216b2d1c53" gracePeriod=30 Sep 11 00:07:42.760446 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:42.760301 2578 generic.go:356] "Generic (PLEG): container finished" podID="3e0c579f-83f1-4f7a-b52f-a71d18df718a" containerID="34b1cf77906f5d84d8b720095071470ec89204586d77995b8920fb28c3284231" exitCode=0 Sep 11 00:07:42.760446 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:42.760341 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k" event={"ID":"3e0c579f-83f1-4f7a-b52f-a71d18df718a","Type":"ContainerDied","Data":"34b1cf77906f5d84d8b720095071470ec89204586d77995b8920fb28c3284231"} Sep 11 00:07:42.837385 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:42.837343 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-37692b-rjbfs"] Sep 11 00:07:42.839811 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:42.839787 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-37692b-rjbfs"] Sep 11 00:07:42.895812 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:42.895788 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k" Sep 11 00:07:42.925881 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:42.925858 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd" Sep 11 00:07:42.959194 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:42.959165 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e0c579f-83f1-4f7a-b52f-a71d18df718a-plugins-config-volume\") pod \"3e0c579f-83f1-4f7a-b52f-a71d18df718a\" (UID: \"3e0c579f-83f1-4f7a-b52f-a71d18df718a\") " Sep 11 00:07:42.959342 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:42.959210 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq25b\" (UniqueName: \"kubernetes.io/projected/3e0c579f-83f1-4f7a-b52f-a71d18df718a-kube-api-access-vq25b\") pod \"3e0c579f-83f1-4f7a-b52f-a71d18df718a\" (UID: \"3e0c579f-83f1-4f7a-b52f-a71d18df718a\") " Sep 11 00:07:42.959518 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:42.959489 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e0c579f-83f1-4f7a-b52f-a71d18df718a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3e0c579f-83f1-4f7a-b52f-a71d18df718a" (UID: "3e0c579f-83f1-4f7a-b52f-a71d18df718a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:07:42.961209 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:42.961179 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e0c579f-83f1-4f7a-b52f-a71d18df718a-kube-api-access-vq25b" (OuterVolumeSpecName: "kube-api-access-vq25b") pod "3e0c579f-83f1-4f7a-b52f-a71d18df718a" (UID: "3e0c579f-83f1-4f7a-b52f-a71d18df718a"). InnerVolumeSpecName "kube-api-access-vq25b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:07:43.044351 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:43.044320 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76dd3efc-c95c-48d4-9910-febb821f1260" path="/var/lib/kubelet/pods/76dd3efc-c95c-48d4-9910-febb821f1260/volumes" Sep 11 00:07:43.044644 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:43.044631 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfbed1b7-c55d-458e-825d-15929e2c5efa" path="/var/lib/kubelet/pods/bfbed1b7-c55d-458e-825d-15929e2c5efa/volumes" Sep 11 00:07:43.059736 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:43.059711 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d53a069-c3ba-4963-b577-fd3035240d4e-plugins-config-volume\") pod \"8d53a069-c3ba-4963-b577-fd3035240d4e\" (UID: \"8d53a069-c3ba-4963-b577-fd3035240d4e\") " Sep 11 00:07:43.059849 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:43.059804 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdnkl\" (UniqueName: \"kubernetes.io/projected/8d53a069-c3ba-4963-b577-fd3035240d4e-kube-api-access-vdnkl\") pod \"8d53a069-c3ba-4963-b577-fd3035240d4e\" (UID: \"8d53a069-c3ba-4963-b577-fd3035240d4e\") " Sep 11 00:07:43.060055 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:43.060038 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e0c579f-83f1-4f7a-b52f-a71d18df718a-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:07:43.060099 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:43.060061 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq25b\" (UniqueName: \"kubernetes.io/projected/3e0c579f-83f1-4f7a-b52f-a71d18df718a-kube-api-access-vq25b\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:07:43.060099 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:43.060062 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d53a069-c3ba-4963-b577-fd3035240d4e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8d53a069-c3ba-4963-b577-fd3035240d4e" (UID: "8d53a069-c3ba-4963-b577-fd3035240d4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:07:43.061796 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:43.061775 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d53a069-c3ba-4963-b577-fd3035240d4e-kube-api-access-vdnkl" (OuterVolumeSpecName: "kube-api-access-vdnkl") pod "8d53a069-c3ba-4963-b577-fd3035240d4e" (UID: "8d53a069-c3ba-4963-b577-fd3035240d4e"). InnerVolumeSpecName "kube-api-access-vdnkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:07:43.160824 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:43.160751 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdnkl\" (UniqueName: \"kubernetes.io/projected/8d53a069-c3ba-4963-b577-fd3035240d4e-kube-api-access-vdnkl\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:07:43.160824 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:43.160776 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d53a069-c3ba-4963-b577-fd3035240d4e-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:07:43.765667 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:43.765633 2578 generic.go:356] "Generic (PLEG): container finished" podID="8d53a069-c3ba-4963-b577-fd3035240d4e" containerID="b3b1001d7dd45ed272552548abb05786d446c6dafbdea89be3bb7d216b2d1c53" exitCode=0 Sep 11 00:07:43.766109 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:43.765708 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd" Sep 11 00:07:43.766109 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:43.765721 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd" event={"ID":"8d53a069-c3ba-4963-b577-fd3035240d4e","Type":"ContainerDied","Data":"b3b1001d7dd45ed272552548abb05786d446c6dafbdea89be3bb7d216b2d1c53"} Sep 11 00:07:43.766109 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:43.765805 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd" event={"ID":"8d53a069-c3ba-4963-b577-fd3035240d4e","Type":"ContainerDied","Data":"4d9cfe6f05994124887993d5e88a2b8d1804ecd4435e8532f56ea4de156dc0ca"} Sep 11 00:07:43.766109 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:43.765830 2578 scope.go:117] "RemoveContainer" containerID="b3b1001d7dd45ed272552548abb05786d446c6dafbdea89be3bb7d216b2d1c53" Sep 11 00:07:43.767282 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:43.767258 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k" event={"ID":"3e0c579f-83f1-4f7a-b52f-a71d18df718a","Type":"ContainerDied","Data":"3db16b4377435307bcc1a9261d43d773fb17ffc70190cbaf6598569e5b5ae10d"} Sep 11 00:07:43.767371 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:43.767297 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k" Sep 11 00:07:43.774401 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:43.774383 2578 scope.go:117] "RemoveContainer" containerID="b3b1001d7dd45ed272552548abb05786d446c6dafbdea89be3bb7d216b2d1c53" Sep 11 00:07:43.774644 ip-10-0-139-142 kubenswrapper[2578]: E0911 00:07:43.774624 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3b1001d7dd45ed272552548abb05786d446c6dafbdea89be3bb7d216b2d1c53\": container with ID starting with b3b1001d7dd45ed272552548abb05786d446c6dafbdea89be3bb7d216b2d1c53 not found: ID does not exist" containerID="b3b1001d7dd45ed272552548abb05786d446c6dafbdea89be3bb7d216b2d1c53" Sep 11 00:07:43.774711 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:43.774650 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3b1001d7dd45ed272552548abb05786d446c6dafbdea89be3bb7d216b2d1c53"} err="failed to get container status \"b3b1001d7dd45ed272552548abb05786d446c6dafbdea89be3bb7d216b2d1c53\": rpc error: code = NotFound desc = could not find container \"b3b1001d7dd45ed272552548abb05786d446c6dafbdea89be3bb7d216b2d1c53\": container with ID starting with b3b1001d7dd45ed272552548abb05786d446c6dafbdea89be3bb7d216b2d1c53 not found: ID does not exist" Sep 11 00:07:43.774711 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:43.774664 2578 scope.go:117] "RemoveContainer" containerID="34b1cf77906f5d84d8b720095071470ec89204586d77995b8920fb28c3284231" Sep 11 00:07:43.782100 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:43.782076 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k"] Sep 11 00:07:43.793043 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:43.793022 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-37692b-6df9c689d9-grn4k"] Sep 11 00:07:43.803758 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:43.803738 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd"] Sep 11 00:07:43.813127 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:43.813107 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-37692b-7f8f7bdc9d-wcnnd"] Sep 11 00:07:45.044123 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:45.044093 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e0c579f-83f1-4f7a-b52f-a71d18df718a" path="/var/lib/kubelet/pods/3e0c579f-83f1-4f7a-b52f-a71d18df718a/volumes" Sep 11 00:07:45.044500 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:45.044443 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d53a069-c3ba-4963-b577-fd3035240d4e" path="/var/lib/kubelet/pods/8d53a069-c3ba-4963-b577-fd3035240d4e/volumes" Sep 11 00:07:49.925105 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:49.925021 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e96d4c-lfc7r"] Sep 11 00:07:49.925538 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:49.925385 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e0c579f-83f1-4f7a-b52f-a71d18df718a" containerName="payload-processing" Sep 11 00:07:49.925538 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:49.925399 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e0c579f-83f1-4f7a-b52f-a71d18df718a" containerName="payload-processing" Sep 11 00:07:49.925538 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:49.925406 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d53a069-c3ba-4963-b577-fd3035240d4e" containerName="payload-pre-processing" Sep 11 00:07:49.925538 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:49.925413 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d53a069-c3ba-4963-b577-fd3035240d4e" containerName="payload-pre-processing" Sep 11 00:07:49.925538 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:49.925437 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfbed1b7-c55d-458e-825d-15929e2c5efa" containerName="revoke-keys" Sep 11 00:07:49.925538 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:49.925442 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfbed1b7-c55d-458e-825d-15929e2c5efa" containerName="revoke-keys" Sep 11 00:07:49.925538 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:49.925490 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d53a069-c3ba-4963-b577-fd3035240d4e" containerName="payload-pre-processing" Sep 11 00:07:49.925538 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:49.925496 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e0c579f-83f1-4f7a-b52f-a71d18df718a" containerName="payload-processing" Sep 11 00:07:49.925538 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:49.925503 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a8d7323-6ca5-439b-b0e0-b060125a130e" containerName="istio-proxy" Sep 11 00:07:49.925538 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:49.925509 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfbed1b7-c55d-458e-825d-15929e2c5efa" containerName="revoke-keys" Sep 11 00:07:49.929756 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:49.929738 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e96d4c-lfc7r" Sep 11 00:07:49.932041 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:49.932021 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-22lrj\"" Sep 11 00:07:49.932157 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:49.932030 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 00:07:49.935882 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:49.935857 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e96d4c-lfc7r"] Sep 11 00:07:50.013905 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:50.013867 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b651af2-e002-47d2-8f6b-8cba583992c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e96d4c-lfc7r\" (UID: \"3b651af2-e002-47d2-8f6b-8cba583992c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e96d4c-lfc7r" Sep 11 00:07:50.114576 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:50.114536 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b651af2-e002-47d2-8f6b-8cba583992c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e96d4c-lfc7r\" (UID: \"3b651af2-e002-47d2-8f6b-8cba583992c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e96d4c-lfc7r" Sep 11 00:07:50.115237 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:50.115214 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b651af2-e002-47d2-8f6b-8cba583992c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e96d4c-lfc7r\" (UID: \"3b651af2-e002-47d2-8f6b-8cba583992c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e96d4c-lfc7r" Sep 11 00:07:50.241045 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:50.240964 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e96d4c-lfc7r" Sep 11 00:07:50.512795 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:50.512770 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e96d4c-lfc7r"] Sep 11 00:07:50.515110 ip-10-0-139-142 kubenswrapper[2578]: W0911 00:07:50.515083 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b651af2_e002_47d2_8f6b_8cba583992c0.slice/crio-b33794c1addac707dbf7bf0bde8848e33a4651704f0b8db2e8cb0c7c4d585cc5 WatchSource:0}: Error finding container b33794c1addac707dbf7bf0bde8848e33a4651704f0b8db2e8cb0c7c4d585cc5: Status 404 returned error can't find the container with id b33794c1addac707dbf7bf0bde8848e33a4651704f0b8db2e8cb0c7c4d585cc5 Sep 11 00:07:50.801539 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:50.801457 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e96d4c-lfc7r" event={"ID":"3b651af2-e002-47d2-8f6b-8cba583992c0","Type":"ContainerStarted","Data":"b33794c1addac707dbf7bf0bde8848e33a4651704f0b8db2e8cb0c7c4d585cc5"} Sep 11 00:07:51.807352 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:51.807317 2578 generic.go:356] "Generic (PLEG): container finished" podID="3b651af2-e002-47d2-8f6b-8cba583992c0" containerID="f6217256b24b025d8151dd3a5603263ae4141c18c826ac9cbc6ccb93d51d906a" exitCode=0 Sep 11 00:07:51.807787 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:51.807401 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e96d4c-lfc7r" event={"ID":"3b651af2-e002-47d2-8f6b-8cba583992c0","Type":"ContainerDied","Data":"f6217256b24b025d8151dd3a5603263ae4141c18c826ac9cbc6ccb93d51d906a"} Sep 11 00:07:52.941413 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:52.941391 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e96d4c-lfc7r" Sep 11 00:07:53.038632 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:53.038602 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b651af2-e002-47d2-8f6b-8cba583992c0-maas-api-service-ca\") pod \"3b651af2-e002-47d2-8f6b-8cba583992c0\" (UID: \"3b651af2-e002-47d2-8f6b-8cba583992c0\") " Sep 11 00:07:53.039013 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:53.038993 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b651af2-e002-47d2-8f6b-8cba583992c0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b651af2-e002-47d2-8f6b-8cba583992c0" (UID: "3b651af2-e002-47d2-8f6b-8cba583992c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:07:53.139277 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:53.139241 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b651af2-e002-47d2-8f6b-8cba583992c0-maas-api-service-ca\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:07:53.819030 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:53.818991 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e96d4c-lfc7r" event={"ID":"3b651af2-e002-47d2-8f6b-8cba583992c0","Type":"ContainerDied","Data":"b33794c1addac707dbf7bf0bde8848e33a4651704f0b8db2e8cb0c7c4d585cc5"} Sep 11 00:07:53.819206 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:53.819051 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b33794c1addac707dbf7bf0bde8848e33a4651704f0b8db2e8cb0c7c4d585cc5" Sep 11 00:07:53.819206 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:07:53.819064 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e96d4c-lfc7r" Sep 11 00:08:00.479815 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:08:00.479781 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw"] Sep 11 00:08:00.480297 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:08:00.480226 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw" podUID="191bd9a2-0920-4f72-9122-bc39ed6f7319" containerName="payload-pre-processing" containerID="cri-o://ea196fdeffbdf7b1e1ec936f8e17fab7b50bd765b1a906ebbcb29ebe2b707464" gracePeriod=30 Sep 11 00:08:00.640688 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:08:00.640650 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e96d4c-lfc7r"] Sep 11 00:08:00.644335 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:08:00.644307 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e96d4c-lfc7r"] Sep 11 00:08:00.723634 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:08:00.723609 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw" Sep 11 00:08:00.852758 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:08:00.852666 2578 generic.go:356] "Generic (PLEG): container finished" podID="191bd9a2-0920-4f72-9122-bc39ed6f7319" containerID="ea196fdeffbdf7b1e1ec936f8e17fab7b50bd765b1a906ebbcb29ebe2b707464" exitCode=0 Sep 11 00:08:00.852758 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:08:00.852707 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw" Sep 11 00:08:00.852758 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:08:00.852749 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw" event={"ID":"191bd9a2-0920-4f72-9122-bc39ed6f7319","Type":"ContainerDied","Data":"ea196fdeffbdf7b1e1ec936f8e17fab7b50bd765b1a906ebbcb29ebe2b707464"} Sep 11 00:08:00.853071 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:08:00.852804 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw" event={"ID":"191bd9a2-0920-4f72-9122-bc39ed6f7319","Type":"ContainerDied","Data":"d4504c3e19ba164cf14e6811a91bc24d5d2724db859913931d9634d33214785b"} Sep 11 00:08:00.853071 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:08:00.852835 2578 scope.go:117] "RemoveContainer" containerID="ea196fdeffbdf7b1e1ec936f8e17fab7b50bd765b1a906ebbcb29ebe2b707464" Sep 11 00:08:00.861132 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:08:00.861116 2578 scope.go:117] "RemoveContainer" containerID="ea196fdeffbdf7b1e1ec936f8e17fab7b50bd765b1a906ebbcb29ebe2b707464" Sep 11 00:08:00.861410 ip-10-0-139-142 kubenswrapper[2578]: E0911 00:08:00.861381 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea196fdeffbdf7b1e1ec936f8e17fab7b50bd765b1a906ebbcb29ebe2b707464\": container with ID starting with ea196fdeffbdf7b1e1ec936f8e17fab7b50bd765b1a906ebbcb29ebe2b707464 not found: ID does not exist" containerID="ea196fdeffbdf7b1e1ec936f8e17fab7b50bd765b1a906ebbcb29ebe2b707464" Sep 11 00:08:00.861514 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:08:00.861418 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea196fdeffbdf7b1e1ec936f8e17fab7b50bd765b1a906ebbcb29ebe2b707464"} err="failed to get container status \"ea196fdeffbdf7b1e1ec936f8e17fab7b50bd765b1a906ebbcb29ebe2b707464\": rpc error: code = NotFound desc = could not find container \"ea196fdeffbdf7b1e1ec936f8e17fab7b50bd765b1a906ebbcb29ebe2b707464\": container with ID starting with ea196fdeffbdf7b1e1ec936f8e17fab7b50bd765b1a906ebbcb29ebe2b707464 not found: ID does not exist" Sep 11 00:08:00.911748 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:08:00.911712 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/191bd9a2-0920-4f72-9122-bc39ed6f7319-plugins-config-volume\") pod \"191bd9a2-0920-4f72-9122-bc39ed6f7319\" (UID: \"191bd9a2-0920-4f72-9122-bc39ed6f7319\") " Sep 11 00:08:00.911910 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:08:00.911762 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwnjh\" (UniqueName: \"kubernetes.io/projected/191bd9a2-0920-4f72-9122-bc39ed6f7319-kube-api-access-zwnjh\") pod \"191bd9a2-0920-4f72-9122-bc39ed6f7319\" (UID: \"191bd9a2-0920-4f72-9122-bc39ed6f7319\") " Sep 11 00:08:00.912134 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:08:00.912110 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191bd9a2-0920-4f72-9122-bc39ed6f7319-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "191bd9a2-0920-4f72-9122-bc39ed6f7319" (UID: "191bd9a2-0920-4f72-9122-bc39ed6f7319"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:08:00.913822 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:08:00.913799 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191bd9a2-0920-4f72-9122-bc39ed6f7319-kube-api-access-zwnjh" (OuterVolumeSpecName: "kube-api-access-zwnjh") pod "191bd9a2-0920-4f72-9122-bc39ed6f7319" (UID: "191bd9a2-0920-4f72-9122-bc39ed6f7319"). InnerVolumeSpecName "kube-api-access-zwnjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:08:01.012967 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:08:01.012914 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/191bd9a2-0920-4f72-9122-bc39ed6f7319-plugins-config-volume\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:08:01.012967 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:08:01.012963 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwnjh\" (UniqueName: \"kubernetes.io/projected/191bd9a2-0920-4f72-9122-bc39ed6f7319-kube-api-access-zwnjh\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:08:01.044565 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:08:01.044533 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b651af2-e002-47d2-8f6b-8cba583992c0" path="/var/lib/kubelet/pods/3b651af2-e002-47d2-8f6b-8cba583992c0/volumes" Sep 11 00:08:01.168839 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:08:01.168801 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw"] Sep 11 00:08:01.175092 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:08:01.175063 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e96d4c-558d576594-75pdw"] Sep 11 00:08:03.043659 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:08:03.043581 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="191bd9a2-0920-4f72-9122-bc39ed6f7319" path="/var/lib/kubelet/pods/191bd9a2-0920-4f72-9122-bc39ed6f7319/volumes" Sep 11 00:08:25.277885 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:08:25.277855 2578 scope.go:117] "RemoveContainer" containerID="e827a7223cf119a4ed20a14be44ee6723ac8c0eaa33a520d24abe089cb6212cd" Sep 11 00:08:25.285449 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:08:25.285431 2578 scope.go:117] "RemoveContainer" containerID="6e4d5d7a62031ca3ba2ff0c6cfb59219c215e526550b48e7389cd91fb749909a" Sep 11 00:08:25.292734 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:08:25.292712 2578 scope.go:117] "RemoveContainer" containerID="2a866841768fca52896465e60934d33a649ac519a6e667542a8cd77c8e573574" Sep 11 00:09:25.329385 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:09:25.329312 2578 scope.go:117] "RemoveContainer" containerID="b28d8abc00285662f8bce7ddbdc47373bf63da26b1fceec1233cbf0791a37604" Sep 11 00:11:25.120628 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:11:25.120594 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zkr6f_aacd921b-5c15-4f7f-8685-cfceebfe5407/console-operator/2.log" Sep 11 00:11:25.124441 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:11:25.124417 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zkr6f_aacd921b-5c15-4f7f-8685-cfceebfe5407/console-operator/2.log" Sep 11 00:11:25.353354 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:11:25.353321 2578 scope.go:117] "RemoveContainer" containerID="44b6e98c79b1b07105f5c6c72c9a2a19b5f0280b4e7ffc62636ce7a66d77b0b2" Sep 11 00:13:21.503258 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:21.503218 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x"] Sep 11 00:13:21.503713 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:21.503506 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="191bd9a2-0920-4f72-9122-bc39ed6f7319" containerName="payload-pre-processing" Sep 11 00:13:21.503713 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:21.503516 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="191bd9a2-0920-4f72-9122-bc39ed6f7319" containerName="payload-pre-processing" Sep 11 00:13:21.503713 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:21.503524 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b651af2-e002-47d2-8f6b-8cba583992c0" containerName="revoke-keys" Sep 11 00:13:21.503713 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:21.503529 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b651af2-e002-47d2-8f6b-8cba583992c0" containerName="revoke-keys" Sep 11 00:13:21.503713 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:21.503587 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="191bd9a2-0920-4f72-9122-bc39ed6f7319" containerName="payload-pre-processing" Sep 11 00:13:21.503713 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:21.503596 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b651af2-e002-47d2-8f6b-8cba583992c0" containerName="revoke-keys" Sep 11 00:13:21.506366 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:21.506350 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x" Sep 11 00:13:21.508681 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:21.508654 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 00:13:21.509628 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:21.509606 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 00:13:21.509738 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:21.509663 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 00:13:21.509738 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:21.509671 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 11 00:13:21.509738 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:21.509659 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-h7jzs\"" Sep 11 00:13:21.516886 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:21.516857 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x"] Sep 11 00:13:21.591926 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:21.591891 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9225a03-6559-4ba2-82ed-4c881d9973a2-metrics-certs\") pod \"maas-api-78cdb675f8-kjz4x\" (UID: \"e9225a03-6559-4ba2-82ed-4c881d9973a2\") " pod="odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x" Sep 11 00:13:21.591926 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:21.591931 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw57d\" (UniqueName: \"kubernetes.io/projected/e9225a03-6559-4ba2-82ed-4c881d9973a2-kube-api-access-xw57d\") pod \"maas-api-78cdb675f8-kjz4x\" (UID: \"e9225a03-6559-4ba2-82ed-4c881d9973a2\") " pod="odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x" Sep 11 00:13:21.592166 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:21.592009 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9225a03-6559-4ba2-82ed-4c881d9973a2-maas-api-tls\") pod \"maas-api-78cdb675f8-kjz4x\" (UID: \"e9225a03-6559-4ba2-82ed-4c881d9973a2\") " pod="odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x" Sep 11 00:13:21.592166 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:21.592024 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9225a03-6559-4ba2-82ed-4c881d9973a2-openshift-service-ca\") pod \"maas-api-78cdb675f8-kjz4x\" (UID: \"e9225a03-6559-4ba2-82ed-4c881d9973a2\") " pod="odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x" Sep 11 00:13:21.692825 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:21.692789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9225a03-6559-4ba2-82ed-4c881d9973a2-metrics-certs\") pod \"maas-api-78cdb675f8-kjz4x\" (UID: \"e9225a03-6559-4ba2-82ed-4c881d9973a2\") " pod="odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x" Sep 11 00:13:21.692825 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:21.692824 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw57d\" (UniqueName: \"kubernetes.io/projected/e9225a03-6559-4ba2-82ed-4c881d9973a2-kube-api-access-xw57d\") pod \"maas-api-78cdb675f8-kjz4x\" (UID: \"e9225a03-6559-4ba2-82ed-4c881d9973a2\") " pod="odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x" Sep 11 00:13:21.693086 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:21.692861 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9225a03-6559-4ba2-82ed-4c881d9973a2-maas-api-tls\") pod \"maas-api-78cdb675f8-kjz4x\" (UID: \"e9225a03-6559-4ba2-82ed-4c881d9973a2\") " pod="odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x" Sep 11 00:13:21.693086 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:21.693048 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9225a03-6559-4ba2-82ed-4c881d9973a2-openshift-service-ca\") pod \"maas-api-78cdb675f8-kjz4x\" (UID: \"e9225a03-6559-4ba2-82ed-4c881d9973a2\") " pod="odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x" Sep 11 00:13:21.693870 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:21.693848 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9225a03-6559-4ba2-82ed-4c881d9973a2-openshift-service-ca\") pod \"maas-api-78cdb675f8-kjz4x\" (UID: \"e9225a03-6559-4ba2-82ed-4c881d9973a2\") " pod="odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x" Sep 11 00:13:21.695422 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:21.695387 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9225a03-6559-4ba2-82ed-4c881d9973a2-maas-api-tls\") pod \"maas-api-78cdb675f8-kjz4x\" (UID: \"e9225a03-6559-4ba2-82ed-4c881d9973a2\") " pod="odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x" Sep 11 00:13:21.695527 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:21.695489 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9225a03-6559-4ba2-82ed-4c881d9973a2-metrics-certs\") pod \"maas-api-78cdb675f8-kjz4x\" (UID: \"e9225a03-6559-4ba2-82ed-4c881d9973a2\") " pod="odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x" Sep 11 00:13:21.700580 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:21.700551 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw57d\" (UniqueName: \"kubernetes.io/projected/e9225a03-6559-4ba2-82ed-4c881d9973a2-kube-api-access-xw57d\") pod \"maas-api-78cdb675f8-kjz4x\" (UID: \"e9225a03-6559-4ba2-82ed-4c881d9973a2\") " pod="odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x" Sep 11 00:13:21.816518 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:21.816426 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x" Sep 11 00:13:22.088877 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:22.088853 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x"] Sep 11 00:13:22.091235 ip-10-0-139-142 kubenswrapper[2578]: W0911 00:13:22.091204 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9225a03_6559_4ba2_82ed_4c881d9973a2.slice/crio-7e1bbd96cdd8255fa69e8fbf7ef0c22e86ebeb926befdda67b98dbd2cccc4139 WatchSource:0}: Error finding container 7e1bbd96cdd8255fa69e8fbf7ef0c22e86ebeb926befdda67b98dbd2cccc4139: Status 404 returned error can't find the container with id 7e1bbd96cdd8255fa69e8fbf7ef0c22e86ebeb926befdda67b98dbd2cccc4139 Sep 11 00:13:22.092486 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:22.092468 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 00:13:22.255136 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:22.255101 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x" event={"ID":"e9225a03-6559-4ba2-82ed-4c881d9973a2","Type":"ContainerStarted","Data":"7e1bbd96cdd8255fa69e8fbf7ef0c22e86ebeb926befdda67b98dbd2cccc4139"} Sep 11 00:13:25.271842 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:25.271799 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x" event={"ID":"e9225a03-6559-4ba2-82ed-4c881d9973a2","Type":"ContainerStarted","Data":"8c387ab4851c0c112e93f8133fdbf9fec9885d714289217a81235fc32f33c295"} Sep 11 00:13:25.272316 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:25.271882 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x" Sep 11 00:13:25.289060 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:25.289014 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x" podStartSLOduration=1.994386556 podStartE2EDuration="4.28899908s" totalImagesPullingTime="2.294612524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:13:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:13:25.287834353 +0000 UTC m=+2220.755308679" watchObservedRunningTime="2026-09-11 00:13:25.28899908 +0000 UTC m=+2220.756473405" Sep 11 00:13:25.375772 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:25.375741 2578 scope.go:117] "RemoveContainer" containerID="6c211b00300a777e4cb5b6ebe2e036d06d152013328013020b7f205d27f5f694" Sep 11 00:13:28.761284 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:28.761249 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x"] Sep 11 00:13:28.761695 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:28.761477 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x" podUID="e9225a03-6559-4ba2-82ed-4c881d9973a2" containerName="maas-api" containerID="cri-o://8c387ab4851c0c112e93f8133fdbf9fec9885d714289217a81235fc32f33c295" gracePeriod=30 Sep 11 00:13:28.794389 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:28.794360 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6664b7b778-6tnqh"] Sep 11 00:13:28.797357 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:28.797341 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6664b7b778-6tnqh" Sep 11 00:13:28.812313 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:28.812275 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6664b7b778-6tnqh"] Sep 11 00:13:28.960510 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:28.960476 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27da97f7-f393-4ef0-a84c-ac8f4a9b146e-metrics-certs\") pod \"maas-api-6664b7b778-6tnqh\" (UID: \"27da97f7-f393-4ef0-a84c-ac8f4a9b146e\") " pod="odh-ai-gateway-infra/maas-api-6664b7b778-6tnqh" Sep 11 00:13:28.960510 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:28.960513 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqrf2\" (UniqueName: \"kubernetes.io/projected/27da97f7-f393-4ef0-a84c-ac8f4a9b146e-kube-api-access-kqrf2\") pod \"maas-api-6664b7b778-6tnqh\" (UID: \"27da97f7-f393-4ef0-a84c-ac8f4a9b146e\") " pod="odh-ai-gateway-infra/maas-api-6664b7b778-6tnqh" Sep 11 00:13:28.960703 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:28.960542 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27da97f7-f393-4ef0-a84c-ac8f4a9b146e-maas-api-tls\") pod \"maas-api-6664b7b778-6tnqh\" (UID: \"27da97f7-f393-4ef0-a84c-ac8f4a9b146e\") " pod="odh-ai-gateway-infra/maas-api-6664b7b778-6tnqh" Sep 11 00:13:28.960741 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:28.960719 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27da97f7-f393-4ef0-a84c-ac8f4a9b146e-openshift-service-ca\") pod \"maas-api-6664b7b778-6tnqh\" (UID: \"27da97f7-f393-4ef0-a84c-ac8f4a9b146e\") " pod="odh-ai-gateway-infra/maas-api-6664b7b778-6tnqh" Sep 11 00:13:29.004789 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.004763 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x" Sep 11 00:13:29.062103 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.062031 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27da97f7-f393-4ef0-a84c-ac8f4a9b146e-maas-api-tls\") pod \"maas-api-6664b7b778-6tnqh\" (UID: \"27da97f7-f393-4ef0-a84c-ac8f4a9b146e\") " pod="odh-ai-gateway-infra/maas-api-6664b7b778-6tnqh" Sep 11 00:13:29.062251 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.062100 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27da97f7-f393-4ef0-a84c-ac8f4a9b146e-openshift-service-ca\") pod \"maas-api-6664b7b778-6tnqh\" (UID: \"27da97f7-f393-4ef0-a84c-ac8f4a9b146e\") " pod="odh-ai-gateway-infra/maas-api-6664b7b778-6tnqh" Sep 11 00:13:29.062251 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.062142 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27da97f7-f393-4ef0-a84c-ac8f4a9b146e-metrics-certs\") pod \"maas-api-6664b7b778-6tnqh\" (UID: \"27da97f7-f393-4ef0-a84c-ac8f4a9b146e\") " pod="odh-ai-gateway-infra/maas-api-6664b7b778-6tnqh" Sep 11 00:13:29.062251 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.062157 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqrf2\" (UniqueName: \"kubernetes.io/projected/27da97f7-f393-4ef0-a84c-ac8f4a9b146e-kube-api-access-kqrf2\") pod \"maas-api-6664b7b778-6tnqh\" (UID: \"27da97f7-f393-4ef0-a84c-ac8f4a9b146e\") " pod="odh-ai-gateway-infra/maas-api-6664b7b778-6tnqh" Sep 11 00:13:29.062834 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.062810 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27da97f7-f393-4ef0-a84c-ac8f4a9b146e-openshift-service-ca\") pod \"maas-api-6664b7b778-6tnqh\" (UID: \"27da97f7-f393-4ef0-a84c-ac8f4a9b146e\") " pod="odh-ai-gateway-infra/maas-api-6664b7b778-6tnqh" Sep 11 00:13:29.064497 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.064473 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27da97f7-f393-4ef0-a84c-ac8f4a9b146e-metrics-certs\") pod \"maas-api-6664b7b778-6tnqh\" (UID: \"27da97f7-f393-4ef0-a84c-ac8f4a9b146e\") " pod="odh-ai-gateway-infra/maas-api-6664b7b778-6tnqh" Sep 11 00:13:29.064593 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.064517 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27da97f7-f393-4ef0-a84c-ac8f4a9b146e-maas-api-tls\") pod \"maas-api-6664b7b778-6tnqh\" (UID: \"27da97f7-f393-4ef0-a84c-ac8f4a9b146e\") " pod="odh-ai-gateway-infra/maas-api-6664b7b778-6tnqh" Sep 11 00:13:29.070129 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.070109 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqrf2\" (UniqueName: \"kubernetes.io/projected/27da97f7-f393-4ef0-a84c-ac8f4a9b146e-kube-api-access-kqrf2\") pod \"maas-api-6664b7b778-6tnqh\" (UID: \"27da97f7-f393-4ef0-a84c-ac8f4a9b146e\") " pod="odh-ai-gateway-infra/maas-api-6664b7b778-6tnqh" Sep 11 00:13:29.136819 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.136789 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6664b7b778-6tnqh" Sep 11 00:13:29.163032 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.163001 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9225a03-6559-4ba2-82ed-4c881d9973a2-maas-api-tls\") pod \"e9225a03-6559-4ba2-82ed-4c881d9973a2\" (UID: \"e9225a03-6559-4ba2-82ed-4c881d9973a2\") " Sep 11 00:13:29.163140 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.163122 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xw57d\" (UniqueName: \"kubernetes.io/projected/e9225a03-6559-4ba2-82ed-4c881d9973a2-kube-api-access-xw57d\") pod \"e9225a03-6559-4ba2-82ed-4c881d9973a2\" (UID: \"e9225a03-6559-4ba2-82ed-4c881d9973a2\") " Sep 11 00:13:29.163187 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.163162 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9225a03-6559-4ba2-82ed-4c881d9973a2-metrics-certs\") pod \"e9225a03-6559-4ba2-82ed-4c881d9973a2\" (UID: \"e9225a03-6559-4ba2-82ed-4c881d9973a2\") " Sep 11 00:13:29.163231 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.163217 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9225a03-6559-4ba2-82ed-4c881d9973a2-openshift-service-ca\") pod \"e9225a03-6559-4ba2-82ed-4c881d9973a2\" (UID: \"e9225a03-6559-4ba2-82ed-4c881d9973a2\") " Sep 11 00:13:29.163646 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.163619 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9225a03-6559-4ba2-82ed-4c881d9973a2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e9225a03-6559-4ba2-82ed-4c881d9973a2" (UID: "e9225a03-6559-4ba2-82ed-4c881d9973a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:13:29.165221 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.165198 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9225a03-6559-4ba2-82ed-4c881d9973a2-kube-api-access-xw57d" (OuterVolumeSpecName: "kube-api-access-xw57d") pod "e9225a03-6559-4ba2-82ed-4c881d9973a2" (UID: "e9225a03-6559-4ba2-82ed-4c881d9973a2"). InnerVolumeSpecName "kube-api-access-xw57d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:13:29.165221 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.165208 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9225a03-6559-4ba2-82ed-4c881d9973a2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e9225a03-6559-4ba2-82ed-4c881d9973a2" (UID: "e9225a03-6559-4ba2-82ed-4c881d9973a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 00:13:29.165583 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.165564 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9225a03-6559-4ba2-82ed-4c881d9973a2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e9225a03-6559-4ba2-82ed-4c881d9973a2" (UID: "e9225a03-6559-4ba2-82ed-4c881d9973a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 00:13:29.265028 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.264992 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9225a03-6559-4ba2-82ed-4c881d9973a2-maas-api-tls\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:13:29.265028 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.265032 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw57d\" (UniqueName: \"kubernetes.io/projected/e9225a03-6559-4ba2-82ed-4c881d9973a2-kube-api-access-xw57d\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:13:29.265213 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.265048 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9225a03-6559-4ba2-82ed-4c881d9973a2-metrics-certs\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:13:29.265213 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.265062 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9225a03-6559-4ba2-82ed-4c881d9973a2-openshift-service-ca\") on node \"ip-10-0-139-142.ec2.internal\" DevicePath \"\"" Sep 11 00:13:29.290706 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.290673 2578 generic.go:356] "Generic (PLEG): container finished" podID="e9225a03-6559-4ba2-82ed-4c881d9973a2" containerID="8c387ab4851c0c112e93f8133fdbf9fec9885d714289217a81235fc32f33c295" exitCode=0 Sep 11 00:13:29.290881 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.290721 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x" Sep 11 00:13:29.290881 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.290716 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x" event={"ID":"e9225a03-6559-4ba2-82ed-4c881d9973a2","Type":"ContainerDied","Data":"8c387ab4851c0c112e93f8133fdbf9fec9885d714289217a81235fc32f33c295"} Sep 11 00:13:29.290881 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.290841 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x" event={"ID":"e9225a03-6559-4ba2-82ed-4c881d9973a2","Type":"ContainerDied","Data":"7e1bbd96cdd8255fa69e8fbf7ef0c22e86ebeb926befdda67b98dbd2cccc4139"} Sep 11 00:13:29.290881 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.290864 2578 scope.go:117] "RemoveContainer" containerID="8c387ab4851c0c112e93f8133fdbf9fec9885d714289217a81235fc32f33c295" Sep 11 00:13:29.299729 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.299708 2578 scope.go:117] "RemoveContainer" containerID="8c387ab4851c0c112e93f8133fdbf9fec9885d714289217a81235fc32f33c295" Sep 11 00:13:29.300024 ip-10-0-139-142 kubenswrapper[2578]: E0911 00:13:29.300005 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c387ab4851c0c112e93f8133fdbf9fec9885d714289217a81235fc32f33c295\": container with ID starting with 8c387ab4851c0c112e93f8133fdbf9fec9885d714289217a81235fc32f33c295 not found: ID does not exist" containerID="8c387ab4851c0c112e93f8133fdbf9fec9885d714289217a81235fc32f33c295" Sep 11 00:13:29.300094 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.300032 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c387ab4851c0c112e93f8133fdbf9fec9885d714289217a81235fc32f33c295"} err="failed to get container status \"8c387ab4851c0c112e93f8133fdbf9fec9885d714289217a81235fc32f33c295\": rpc error: code = NotFound desc = could not find container \"8c387ab4851c0c112e93f8133fdbf9fec9885d714289217a81235fc32f33c295\": container with ID starting with 8c387ab4851c0c112e93f8133fdbf9fec9885d714289217a81235fc32f33c295 not found: ID does not exist" Sep 11 00:13:29.313789 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.313733 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x"] Sep 11 00:13:29.317931 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.317908 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-78cdb675f8-kjz4x"] Sep 11 00:13:29.406634 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:29.406609 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6664b7b778-6tnqh"] Sep 11 00:13:29.408524 ip-10-0-139-142 kubenswrapper[2578]: W0911 00:13:29.408496 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27da97f7_f393_4ef0_a84c_ac8f4a9b146e.slice/crio-a49ccfa725e95f6c4659fcee870359dcf232e68cb5407d13416bb5ac00989ec3 WatchSource:0}: Error finding container a49ccfa725e95f6c4659fcee870359dcf232e68cb5407d13416bb5ac00989ec3: Status 404 returned error can't find the container with id a49ccfa725e95f6c4659fcee870359dcf232e68cb5407d13416bb5ac00989ec3 Sep 11 00:13:30.296762 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:30.296723 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6664b7b778-6tnqh" event={"ID":"27da97f7-f393-4ef0-a84c-ac8f4a9b146e","Type":"ContainerStarted","Data":"8d549524dd6376bbcb2490208ad646b6bf509247b07785369828ac2d7b9a24e6"} Sep 11 00:13:30.297229 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:30.296884 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6664b7b778-6tnqh" Sep 11 00:13:30.297229 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:30.296924 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6664b7b778-6tnqh" event={"ID":"27da97f7-f393-4ef0-a84c-ac8f4a9b146e","Type":"ContainerStarted","Data":"a49ccfa725e95f6c4659fcee870359dcf232e68cb5407d13416bb5ac00989ec3"} Sep 11 00:13:30.314134 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:30.314087 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6664b7b778-6tnqh" podStartSLOduration=1.911865903 podStartE2EDuration="2.314072007s" totalImagesPullingTime="402.206104ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:13:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:13:30.311903036 +0000 UTC m=+2225.779377362" watchObservedRunningTime="2026-09-11 00:13:30.314072007 +0000 UTC m=+2225.781546331" Sep 11 00:13:31.044320 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:31.044288 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9225a03-6559-4ba2-82ed-4c881d9973a2" path="/var/lib/kubelet/pods/e9225a03-6559-4ba2-82ed-4c881d9973a2/volumes" Sep 11 00:13:36.308127 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:13:36.308095 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6664b7b778-6tnqh" Sep 11 00:14:01.505331 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:01.505243 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-94j5h_8860c823-1800-441d-8c50-4c10e31e99aa/manager/0.log" Sep 11 00:14:01.857817 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:01.857744 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f676b6c5d-z9t4c_846a18d9-8a3f-4e26-abbc-6d39a681cd3d/copy-manifests/0.log" Sep 11 00:14:01.875710 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:01.875683 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f676b6c5d-z9t4c_846a18d9-8a3f-4e26-abbc-6d39a681cd3d/manager/0.log" Sep 11 00:14:02.196958 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:02.196907 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6664b7b778-6tnqh_27da97f7-f393-4ef0-a84c-ac8f4a9b146e/maas-api/0.log" Sep 11 00:14:04.327045 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:04.326983 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m_9a705061-7770-4fae-a453-dd7491b000e6/istio-proxy/0.log" Sep 11 00:14:04.449046 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:04.449012 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-hh444_4acfeb77-59ca-4083-8f4b-062f47030c56/discovery/0.log" Sep 11 00:14:04.909258 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:04.909223 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fkscd_2b88901f-6268-423c-b4d2-8d5a44cdb607/payload-pre-processing/0.log" Sep 11 00:14:05.482860 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:05.482827 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-52jr6_d2efb490-ac69-4c42-a41e-7c98483436f8/main/0.log" Sep 11 00:14:05.696990 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:05.696951 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-vrtdb_42570555-7956-4ee2-9c38-0c86484711ef/main/0.log" Sep 11 00:14:05.913673 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:05.913640 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5kvnl_6ebaa8c0-7bb2-4bbc-b351-92c3cb89d3ab/main/0.log" Sep 11 00:14:15.196645 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:15.196609 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zzfhc_9944f5db-ef8d-4f9a-86a8-a7c4a3838cb7/global-pull-secret-syncer/0.log" Sep 11 00:14:15.278767 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:15.278729 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kr8fb_4b358463-cfe2-49ec-91ea-64ae0856ccd7/konnectivity-agent/0.log" Sep 11 00:14:15.345023 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:15.344989 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-142.ec2.internal_bff3cfe4b5e3fbd1418d844a741d3b63/haproxy/0.log" Sep 11 00:14:21.955356 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:21.955322 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-crtrr_4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72/node-exporter/0.log" Sep 11 00:14:21.981746 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:21.981715 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-crtrr_4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72/kube-rbac-proxy/0.log" Sep 11 00:14:22.000379 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:22.000351 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-crtrr_4b5a2d4f-1293-4bb0-bdbb-a67a3dca3e72/init-textfile/0.log" Sep 11 00:14:24.864187 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:24.864155 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zkr6f_aacd921b-5c15-4f7f-8685-cfceebfe5407/console-operator/2.log" Sep 11 00:14:24.871440 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:24.871410 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-zkr6f_aacd921b-5c15-4f7f-8685-cfceebfe5407/console-operator/3.log" Sep 11 00:14:25.392529 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:25.392498 2578 scope.go:117] "RemoveContainer" containerID="f6217256b24b025d8151dd3a5603263ae4141c18c826ac9cbc6ccb93d51d906a" Sep 11 00:14:25.400459 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:25.400436 2578 scope.go:117] "RemoveContainer" containerID="4b500013dff32e4adefeaf914f2fd3d0eff9de39dc4a9e4df5465bae2fbfbd5a" Sep 11 00:14:25.648974 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:25.648858 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-szhzc/perf-node-gather-daemonset-bthpn"] Sep 11 00:14:25.649598 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:25.649574 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9225a03-6559-4ba2-82ed-4c881d9973a2" containerName="maas-api" Sep 11 00:14:25.649709 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:25.649601 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9225a03-6559-4ba2-82ed-4c881d9973a2" containerName="maas-api" Sep 11 00:14:25.649764 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:25.649736 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9225a03-6559-4ba2-82ed-4c881d9973a2" containerName="maas-api" Sep 11 00:14:25.655012 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:25.654986 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-szhzc/perf-node-gather-daemonset-bthpn" Sep 11 00:14:25.657391 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:25.657365 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-szhzc\"/\"kube-root-ca.crt\"" Sep 11 00:14:25.658178 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:25.658154 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-szhzc\"/\"openshift-service-ca.crt\"" Sep 11 00:14:25.658178 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:25.658171 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-szhzc\"/\"default-dockercfg-fm6hk\"" Sep 11 00:14:25.660126 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:25.660101 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-szhzc/perf-node-gather-daemonset-bthpn"] Sep 11 00:14:25.756245 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:25.756209 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21c7fb42-fe23-4540-b168-841ca28ba4c0-lib-modules\") pod \"perf-node-gather-daemonset-bthpn\" (UID: \"21c7fb42-fe23-4540-b168-841ca28ba4c0\") " pod="openshift-must-gather-szhzc/perf-node-gather-daemonset-bthpn" Sep 11 00:14:25.756444 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:25.756270 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbhfr\" (UniqueName: \"kubernetes.io/projected/21c7fb42-fe23-4540-b168-841ca28ba4c0-kube-api-access-qbhfr\") pod \"perf-node-gather-daemonset-bthpn\" (UID: \"21c7fb42-fe23-4540-b168-841ca28ba4c0\") " pod="openshift-must-gather-szhzc/perf-node-gather-daemonset-bthpn" Sep 11 00:14:25.756444 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:25.756313 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/21c7fb42-fe23-4540-b168-841ca28ba4c0-proc\") pod \"perf-node-gather-daemonset-bthpn\" (UID: \"21c7fb42-fe23-4540-b168-841ca28ba4c0\") " pod="openshift-must-gather-szhzc/perf-node-gather-daemonset-bthpn" Sep 11 00:14:25.756444 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:25.756351 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/21c7fb42-fe23-4540-b168-841ca28ba4c0-podres\") pod \"perf-node-gather-daemonset-bthpn\" (UID: \"21c7fb42-fe23-4540-b168-841ca28ba4c0\") " pod="openshift-must-gather-szhzc/perf-node-gather-daemonset-bthpn" Sep 11 00:14:25.756444 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:25.756380 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21c7fb42-fe23-4540-b168-841ca28ba4c0-sys\") pod \"perf-node-gather-daemonset-bthpn\" (UID: \"21c7fb42-fe23-4540-b168-841ca28ba4c0\") " pod="openshift-must-gather-szhzc/perf-node-gather-daemonset-bthpn" Sep 11 00:14:25.857509 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:25.857469 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/21c7fb42-fe23-4540-b168-841ca28ba4c0-podres\") pod \"perf-node-gather-daemonset-bthpn\" (UID: \"21c7fb42-fe23-4540-b168-841ca28ba4c0\") " pod="openshift-must-gather-szhzc/perf-node-gather-daemonset-bthpn" Sep 11 00:14:25.857700 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:25.857521 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21c7fb42-fe23-4540-b168-841ca28ba4c0-sys\") pod \"perf-node-gather-daemonset-bthpn\" (UID: \"21c7fb42-fe23-4540-b168-841ca28ba4c0\") " pod="openshift-must-gather-szhzc/perf-node-gather-daemonset-bthpn" Sep 11 00:14:25.857700 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:25.857568 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21c7fb42-fe23-4540-b168-841ca28ba4c0-lib-modules\") pod \"perf-node-gather-daemonset-bthpn\" (UID: \"21c7fb42-fe23-4540-b168-841ca28ba4c0\") " pod="openshift-must-gather-szhzc/perf-node-gather-daemonset-bthpn" Sep 11 00:14:25.857700 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:25.857623 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbhfr\" (UniqueName: \"kubernetes.io/projected/21c7fb42-fe23-4540-b168-841ca28ba4c0-kube-api-access-qbhfr\") pod \"perf-node-gather-daemonset-bthpn\" (UID: \"21c7fb42-fe23-4540-b168-841ca28ba4c0\") " pod="openshift-must-gather-szhzc/perf-node-gather-daemonset-bthpn" Sep 11 00:14:25.857700 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:25.857640 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/21c7fb42-fe23-4540-b168-841ca28ba4c0-podres\") pod \"perf-node-gather-daemonset-bthpn\" (UID: \"21c7fb42-fe23-4540-b168-841ca28ba4c0\") " pod="openshift-must-gather-szhzc/perf-node-gather-daemonset-bthpn" Sep 11 00:14:25.857700 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:25.857657 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/21c7fb42-fe23-4540-b168-841ca28ba4c0-proc\") pod \"perf-node-gather-daemonset-bthpn\" (UID: \"21c7fb42-fe23-4540-b168-841ca28ba4c0\") " pod="openshift-must-gather-szhzc/perf-node-gather-daemonset-bthpn" Sep 11 00:14:25.857700 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:25.857657 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21c7fb42-fe23-4540-b168-841ca28ba4c0-sys\") pod \"perf-node-gather-daemonset-bthpn\" (UID: \"21c7fb42-fe23-4540-b168-841ca28ba4c0\") " pod="openshift-must-gather-szhzc/perf-node-gather-daemonset-bthpn" Sep 11 00:14:25.858059 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:25.857709 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/21c7fb42-fe23-4540-b168-841ca28ba4c0-proc\") pod \"perf-node-gather-daemonset-bthpn\" (UID: \"21c7fb42-fe23-4540-b168-841ca28ba4c0\") " pod="openshift-must-gather-szhzc/perf-node-gather-daemonset-bthpn" Sep 11 00:14:25.858059 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:25.857731 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21c7fb42-fe23-4540-b168-841ca28ba4c0-lib-modules\") pod \"perf-node-gather-daemonset-bthpn\" (UID: \"21c7fb42-fe23-4540-b168-841ca28ba4c0\") " pod="openshift-must-gather-szhzc/perf-node-gather-daemonset-bthpn" Sep 11 00:14:25.865862 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:25.865840 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbhfr\" (UniqueName: \"kubernetes.io/projected/21c7fb42-fe23-4540-b168-841ca28ba4c0-kube-api-access-qbhfr\") pod \"perf-node-gather-daemonset-bthpn\" (UID: \"21c7fb42-fe23-4540-b168-841ca28ba4c0\") " pod="openshift-must-gather-szhzc/perf-node-gather-daemonset-bthpn" Sep 11 00:14:25.965679 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:25.965586 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-szhzc/perf-node-gather-daemonset-bthpn" Sep 11 00:14:26.235557 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:26.235527 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-szhzc/perf-node-gather-daemonset-bthpn"] Sep 11 00:14:26.237142 ip-10-0-139-142 kubenswrapper[2578]: W0911 00:14:26.237108 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod21c7fb42_fe23_4540_b168_841ca28ba4c0.slice/crio-db2414fb3c32e9a9a135b865f2a375cf996e29b212941792ca3b9b32881e8e0f WatchSource:0}: Error finding container db2414fb3c32e9a9a135b865f2a375cf996e29b212941792ca3b9b32881e8e0f: Status 404 returned error can't find the container with id db2414fb3c32e9a9a135b865f2a375cf996e29b212941792ca3b9b32881e8e0f Sep 11 00:14:26.545035 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:26.544929 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-szhzc/perf-node-gather-daemonset-bthpn" event={"ID":"21c7fb42-fe23-4540-b168-841ca28ba4c0","Type":"ContainerStarted","Data":"bb67c486984ba8c46bc0a90f154276371f678c76df02edb78a53578dfa932b5d"} Sep 11 00:14:26.545035 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:26.545002 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-szhzc/perf-node-gather-daemonset-bthpn" event={"ID":"21c7fb42-fe23-4540-b168-841ca28ba4c0","Type":"ContainerStarted","Data":"db2414fb3c32e9a9a135b865f2a375cf996e29b212941792ca3b9b32881e8e0f"} Sep 11 00:14:26.545216 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:26.545058 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-szhzc/perf-node-gather-daemonset-bthpn" Sep 11 00:14:26.561115 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:26.561063 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-szhzc/perf-node-gather-daemonset-bthpn" podStartSLOduration=1.561048944 podStartE2EDuration="1.561048944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:14:26.560566887 +0000 UTC m=+2282.028041242" watchObservedRunningTime="2026-09-11 00:14:26.561048944 +0000 UTC m=+2282.028523269" Sep 11 00:14:27.444479 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:27.444452 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tmscj_6b74b357-db9d-4962-9eda-ff644ef6f739/dns/0.log" Sep 11 00:14:27.476762 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:27.476717 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tmscj_6b74b357-db9d-4962-9eda-ff644ef6f739/kube-rbac-proxy/0.log" Sep 11 00:14:27.590513 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:27.590479 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dm6sw_1da41227-eab5-43fa-a97c-69f7d935cc76/dns-node-resolver/0.log" Sep 11 00:14:28.251814 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:28.251627 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f7fjr_7be68ac6-c59b-4afa-9974-a39f8067c1f7/node-ca/0.log" Sep 11 00:14:29.194237 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:29.194199 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m_9a705061-7770-4fae-a453-dd7491b000e6/istio-proxy/0.log" Sep 11 00:14:29.597956 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:29.597847 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-hh444_4acfeb77-59ca-4083-8f4b-062f47030c56/discovery/0.log" Sep 11 00:14:29.720233 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:29.720195 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fkscd_2b88901f-6268-423c-b4d2-8d5a44cdb607/payload-pre-processing/0.log" Sep 11 00:14:30.627193 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:30.627164 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-77hbh_0cb537d8-167e-4bcc-9681-1bdc99fd9cef/serve-healthcheck-canary/0.log" Sep 11 00:14:31.123862 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:31.123741 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-gbhmg_b3a36399-90fc-4dbe-86f4-ffc633cd89cc/insights-operator/0.log" Sep 11 00:14:31.124417 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:31.124397 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-gbhmg_b3a36399-90fc-4dbe-86f4-ffc633cd89cc/insights-operator/1.log" Sep 11 00:14:31.144287 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:31.144260 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4l5c2_79b487d6-5e92-45eb-b8fb-03abd1f439b8/kube-rbac-proxy/0.log" Sep 11 00:14:31.163741 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:31.163715 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4l5c2_79b487d6-5e92-45eb-b8fb-03abd1f439b8/exporter/0.log" Sep 11 00:14:31.183125 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:31.183103 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4l5c2_79b487d6-5e92-45eb-b8fb-03abd1f439b8/extractor/0.log" Sep 11 00:14:32.559960 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:32.559912 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-szhzc/perf-node-gather-daemonset-bthpn" Sep 11 00:14:33.883018 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:33.882983 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-94j5h_8860c823-1800-441d-8c50-4c10e31e99aa/manager/0.log" Sep 11 00:14:34.085131 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:34.085091 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f676b6c5d-z9t4c_846a18d9-8a3f-4e26-abbc-6d39a681cd3d/manager/0.log" Sep 11 00:14:34.099870 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:34.099846 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f676b6c5d-z9t4c_846a18d9-8a3f-4e26-abbc-6d39a681cd3d/copy-manifests/0.log" Sep 11 00:14:35.607821 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:35.607786 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-f59994447-lbf89_9b281584-acde-4df9-8810-37383aaf00d7/manager/0.log" Sep 11 00:14:36.746093 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:36.746064 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m_9a705061-7770-4fae-a453-dd7491b000e6/istio-proxy/0.log" Sep 11 00:14:37.148929 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:37.148902 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-hh444_4acfeb77-59ca-4083-8f4b-062f47030c56/discovery/0.log" Sep 11 00:14:37.248534 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:37.248493 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fkscd_2b88901f-6268-423c-b4d2-8d5a44cdb607/payload-pre-processing/0.log" Sep 11 00:14:38.834229 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:38.834198 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-nhw9m_9a705061-7770-4fae-a453-dd7491b000e6/istio-proxy/0.log" Sep 11 00:14:39.254228 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:39.254190 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-hh444_4acfeb77-59ca-4083-8f4b-062f47030c56/discovery/0.log" Sep 11 00:14:39.355570 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:39.355535 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fkscd_2b88901f-6268-423c-b4d2-8d5a44cdb607/payload-pre-processing/0.log" Sep 11 00:14:42.631873 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:42.631845 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-whckt_4580695f-ec27-45ad-9670-4849d9354b05/kube-multus-additional-cni-plugins/0.log" Sep 11 00:14:42.650057 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:42.650032 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-whckt_4580695f-ec27-45ad-9670-4849d9354b05/egress-router-binary-copy/0.log" Sep 11 00:14:42.668983 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:42.668958 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-whckt_4580695f-ec27-45ad-9670-4849d9354b05/cni-plugins/0.log" Sep 11 00:14:42.687212 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:42.687182 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-whckt_4580695f-ec27-45ad-9670-4849d9354b05/bond-cni-plugin/0.log" Sep 11 00:14:42.711125 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:42.711096 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-whckt_4580695f-ec27-45ad-9670-4849d9354b05/routeoverride-cni/0.log" Sep 11 00:14:42.730413 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:42.730393 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-whckt_4580695f-ec27-45ad-9670-4849d9354b05/whereabouts-cni-bincopy/0.log" Sep 11 00:14:42.748588 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:42.748565 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-whckt_4580695f-ec27-45ad-9670-4849d9354b05/whereabouts-cni/0.log" Sep 11 00:14:42.840730 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:42.840702 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-v77rf_4e1b3ec6-1e96-4306-ac02-5aa1a56d37bd/kube-multus/0.log" Sep 11 00:14:42.961922 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:42.961839 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fmvxw_a06f8008-28ac-4cb9-8310-f5c42b169d10/network-metrics-daemon/0.log" Sep 11 00:14:42.977745 ip-10-0-139-142 kubenswrapper[2578]: I0911 00:14:42.977710 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fmvxw_a06f8008-28ac-4cb9-8310-f5c42b169d10/kube-rbac-proxy/0.log"