Sep 11 14:40:08.636023 ip-10-0-135-39 systemd[1]: Starting Kubernetes Kubelet... Sep 11 14:40:09.035459 ip-10-0-135-39 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 14:40:09.035459 ip-10-0-135-39 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 11 14:40:09.035459 ip-10-0-135-39 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 14:40:09.035459 ip-10-0-135-39 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 11 14:40:09.035459 ip-10-0-135-39 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 14:40:09.037712 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.037623 2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 11 14:40:09.040477 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040460 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:40:09.040477 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040476 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:40:09.040546 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040480 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:40:09.040546 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040483 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:40:09.040546 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040486 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:40:09.040546 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040488 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:40:09.040546 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040491 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:40:09.040546 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040496 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:40:09.040546 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040504 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:40:09.040546 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040507 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:40:09.040546 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040510 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:40:09.040546 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040513 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:40:09.040546 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040515 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:40:09.040546 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040518 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:40:09.040546 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040520 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:40:09.040546 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040523 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:40:09.040546 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040526 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:40:09.040546 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040529 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:40:09.040546 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040532 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:40:09.040546 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040535 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:40:09.040546 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040537 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:40:09.041001 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040540 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:40:09.041001 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040543 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:40:09.041001 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040545 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:40:09.041001 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040548 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:40:09.041001 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040551 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:40:09.041001 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040554 2583 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:40:09.041001 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040557 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:40:09.041001 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040560 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:40:09.041001 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040562 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:40:09.041001 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040565 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:40:09.041001 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040567 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:40:09.041001 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040570 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:40:09.041001 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040572 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:40:09.041001 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040575 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:40:09.041001 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040578 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:40:09.041001 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040582 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:40:09.041001 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040585 2583 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:40:09.041001 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040588 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:40:09.041001 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040591 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:40:09.041001 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040594 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:40:09.041501 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040597 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:40:09.041501 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040601 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:40:09.041501 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040605 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:40:09.041501 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040608 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:40:09.041501 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040611 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:40:09.041501 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040614 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:40:09.041501 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040617 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:40:09.041501 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040619 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:40:09.041501 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040622 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:40:09.041501 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040625 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:40:09.041501 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040628 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:40:09.041501 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040630 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:40:09.041501 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040633 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:40:09.041501 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040636 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:40:09.041501 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040638 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:40:09.041501 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040641 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:40:09.041501 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040644 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:40:09.041501 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040646 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:40:09.041501 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040649 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:40:09.041975 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040652 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:40:09.041975 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040655 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:40:09.041975 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040657 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:40:09.041975 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040660 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:40:09.041975 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040662 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:40:09.041975 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040665 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:40:09.041975 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040667 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:40:09.041975 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040670 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:40:09.041975 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040672 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:40:09.041975 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040675 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:40:09.041975 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040678 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:40:09.041975 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040681 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:40:09.041975 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040683 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:40:09.041975 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040686 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:40:09.041975 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040688 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:40:09.041975 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040692 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:40:09.041975 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040694 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:40:09.041975 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040697 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:40:09.041975 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040699 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:40:09.042437 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040716 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:40:09.042437 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040719 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:40:09.042437 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040721 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:40:09.042437 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040724 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:40:09.042437 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040727 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:40:09.042437 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040730 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:40:09.042437 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040733 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:40:09.042437 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040736 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:40:09.042437 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040739 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:40:09.042437 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040741 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:40:09.042437 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040744 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:40:09.042437 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040746 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:40:09.042437 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040749 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:40:09.042437 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040752 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:40:09.042437 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040754 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:40:09.042437 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040757 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:40:09.042437 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040759 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:40:09.042437 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040762 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:40:09.042437 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040764 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:40:09.042437 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.040767 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:40:09.042922 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041468 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:40:09.042922 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041475 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:40:09.042922 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041478 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:40:09.042922 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041481 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:40:09.042922 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041484 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:40:09.042922 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041488 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:40:09.042922 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041490 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:40:09.042922 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041493 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:40:09.042922 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041496 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:40:09.042922 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041498 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:40:09.042922 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041501 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:40:09.042922 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041504 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:40:09.042922 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041507 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:40:09.042922 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041510 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:40:09.042922 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041514 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:40:09.042922 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041517 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:40:09.042922 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041519 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:40:09.042922 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041522 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:40:09.042922 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041525 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:40:09.042922 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041527 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:40:09.043403 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041530 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:40:09.043403 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041532 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:40:09.043403 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041535 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:40:09.043403 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041538 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:40:09.043403 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041540 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:40:09.043403 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041543 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:40:09.043403 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041545 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:40:09.043403 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041548 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:40:09.043403 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041550 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:40:09.043403 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041553 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:40:09.043403 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041555 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:40:09.043403 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041558 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:40:09.043403 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041560 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:40:09.043403 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041563 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:40:09.043403 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041566 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:40:09.043403 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041568 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:40:09.043403 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041573 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:40:09.043403 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041576 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:40:09.043403 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041580 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:40:09.043883 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041583 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:40:09.043883 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041587 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:40:09.043883 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041590 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:40:09.043883 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041593 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:40:09.043883 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041596 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:40:09.043883 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041599 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:40:09.043883 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041603 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:40:09.043883 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041606 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:40:09.043883 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041608 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:40:09.043883 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041611 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:40:09.043883 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041615 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:40:09.043883 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041618 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:40:09.043883 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041621 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:40:09.043883 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041624 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:40:09.043883 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041627 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:40:09.043883 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041630 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:40:09.043883 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041632 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:40:09.043883 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041635 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:40:09.043883 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041637 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:40:09.044355 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041640 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:40:09.044355 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041642 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:40:09.044355 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041645 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:40:09.044355 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041647 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:40:09.044355 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041650 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:40:09.044355 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041652 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:40:09.044355 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041655 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:40:09.044355 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041658 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:40:09.044355 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041661 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:40:09.044355 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041664 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:40:09.044355 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041667 2583 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:40:09.044355 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041669 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:40:09.044355 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041672 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:40:09.044355 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041675 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:40:09.044355 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041677 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:40:09.044355 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041680 2583 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:40:09.044355 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041682 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:40:09.044355 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041685 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:40:09.044355 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041688 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:40:09.044355 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041690 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:40:09.044845 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041693 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:40:09.044845 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041695 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:40:09.044845 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041698 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:40:09.044845 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041716 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:40:09.044845 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041720 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:40:09.044845 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041722 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:40:09.044845 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041725 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:40:09.044845 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041728 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:40:09.044845 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041730 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:40:09.044845 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041733 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:40:09.044845 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041736 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:40:09.044845 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041739 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:40:09.044845 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041741 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:40:09.044845 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041744 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:40:09.044845 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041746 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:40:09.044845 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041749 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:40:09.044845 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041751 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:40:09.044845 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041754 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:40:09.044845 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041756 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:40:09.044845 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041759 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:40:09.045380 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.041762 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:40:09.045380 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042121 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:40:09.045380 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042125 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:40:09.045380 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042128 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:40:09.045380 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042131 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:40:09.045380 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042134 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:40:09.045380 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042137 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:40:09.045380 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042140 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:40:09.045380 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042143 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:40:09.045380 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042147 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:40:09.045380 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042150 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:40:09.045380 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042152 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:40:09.045380 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042155 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:40:09.045380 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042160 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:40:09.045380 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042163 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:40:09.045380 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042166 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:40:09.045380 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042169 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:40:09.045380 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042172 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:40:09.045380 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042175 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:40:09.045858 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042177 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:40:09.045858 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042180 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:40:09.045858 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042183 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:40:09.045858 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042186 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:40:09.045858 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042188 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:40:09.045858 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042191 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:40:09.045858 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042193 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:40:09.045858 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042196 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:40:09.045858 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042198 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:40:09.045858 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042201 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:40:09.045858 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042203 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:40:09.045858 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042206 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:40:09.045858 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042209 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:40:09.045858 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042211 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:40:09.045858 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042214 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:40:09.045858 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042216 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:40:09.045858 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042219 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:40:09.045858 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042222 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:40:09.045858 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042224 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:40:09.045858 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042227 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:40:09.046339 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042230 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:40:09.046339 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042233 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:40:09.046339 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042235 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:40:09.046339 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042238 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:40:09.046339 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042240 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:40:09.046339 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042243 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:40:09.046339 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042246 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:40:09.046339 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042248 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:40:09.046339 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042250 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:40:09.046339 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042253 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:40:09.046339 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042256 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:40:09.046339 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042258 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:40:09.046339 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042261 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:40:09.046339 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042263 2583 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:40:09.046339 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042266 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:40:09.046339 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042269 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:40:09.046339 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042272 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:40:09.046339 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042274 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:40:09.046339 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042277 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:40:09.046339 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042280 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:40:09.046873 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042282 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:40:09.046873 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042285 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:40:09.046873 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042288 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:40:09.046873 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042290 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:40:09.046873 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042293 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:40:09.046873 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042295 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:40:09.046873 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042298 2583 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:40:09.046873 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042301 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:40:09.046873 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042303 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:40:09.046873 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042306 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:40:09.046873 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042309 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:40:09.046873 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042314 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:40:09.046873 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042317 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:40:09.046873 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042321 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:40:09.046873 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042325 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:40:09.046873 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042328 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:40:09.046873 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042331 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:40:09.046873 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042333 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:40:09.046873 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042336 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:40:09.046873 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042339 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:40:09.047430 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042342 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:40:09.047430 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042345 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:40:09.047430 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042348 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:40:09.047430 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042350 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:40:09.047430 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042353 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:40:09.047430 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042355 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:40:09.047430 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042358 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:40:09.047430 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042362 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:40:09.047430 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042365 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:40:09.047430 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042367 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:40:09.047430 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042370 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:40:09.047430 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042373 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:40:09.047430 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042376 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:40:09.047430 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042378 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:40:09.047430 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042381 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:40:09.047430 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042384 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:40:09.047430 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042388 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:40:09.047430 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042391 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:40:09.047430 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042393 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:40:09.047430 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042396 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:40:09.047937 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.042399 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:40:09.047937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042480 2583 flags.go:64] FLAG: --address="0.0.0.0" Sep 11 14:40:09.047937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042518 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 11 14:40:09.047937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042526 2583 flags.go:64] FLAG: --anonymous-auth="true" Sep 11 14:40:09.047937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042530 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 11 14:40:09.047937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042537 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 11 14:40:09.047937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042540 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 11 14:40:09.047937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042544 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 11 14:40:09.047937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042549 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 11 14:40:09.047937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042552 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 11 14:40:09.047937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042555 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 11 14:40:09.047937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042558 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 11 14:40:09.047937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042561 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 11 14:40:09.047937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042565 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 11 14:40:09.047937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042567 2583 flags.go:64] FLAG: --cgroup-root="" Sep 11 14:40:09.047937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042570 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 11 14:40:09.047937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042573 2583 flags.go:64] FLAG: --client-ca-file="" Sep 11 14:40:09.047937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042576 2583 flags.go:64] FLAG: --cloud-provider="external" Sep 11 14:40:09.047937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042578 2583 flags.go:64] FLAG: --cluster-dns="[]" Sep 11 14:40:09.047937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042583 2583 flags.go:64] FLAG: --cluster-domain="" Sep 11 14:40:09.047937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042586 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 11 14:40:09.047937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042589 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 11 14:40:09.047937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042592 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 11 14:40:09.047937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042596 2583 flags.go:64] FLAG: --container-log-max-files="5" Sep 11 14:40:09.048533 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042599 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 11 14:40:09.048533 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042602 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 11 14:40:09.048533 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042605 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 11 14:40:09.048533 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042609 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 11 14:40:09.048533 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042614 2583 flags.go:64] FLAG: --contention-profiling="false" Sep 11 14:40:09.048533 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042617 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 11 14:40:09.048533 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042620 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 11 14:40:09.048533 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042623 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 11 14:40:09.048533 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042625 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 11 14:40:09.048533 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042629 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 11 14:40:09.048533 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042632 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 11 14:40:09.048533 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042638 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 11 14:40:09.048533 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042641 2583 flags.go:64] FLAG: --enable-load-reader="false" Sep 11 14:40:09.048533 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042644 2583 flags.go:64] FLAG: --enable-server="true" Sep 11 14:40:09.048533 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042647 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 11 14:40:09.048533 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042651 2583 flags.go:64] FLAG: --event-burst="100" Sep 11 14:40:09.048533 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042654 2583 flags.go:64] FLAG: --event-qps="50" Sep 11 14:40:09.048533 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042657 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 11 14:40:09.048533 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042660 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 11 14:40:09.048533 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042663 2583 flags.go:64] FLAG: --eviction-hard="" Sep 11 14:40:09.048533 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042667 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 11 14:40:09.048533 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042669 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 11 14:40:09.048533 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042672 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 11 14:40:09.048533 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042675 2583 flags.go:64] FLAG: --eviction-soft="" Sep 11 14:40:09.048533 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042678 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 11 14:40:09.049122 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042681 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 11 14:40:09.049122 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042684 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 11 14:40:09.049122 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042686 2583 flags.go:64] FLAG: --experimental-mounter-path="" Sep 11 14:40:09.049122 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042689 2583 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 11 14:40:09.049122 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042692 2583 flags.go:64] FLAG: --fail-swap-on="true" Sep 11 14:40:09.049122 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042695 2583 flags.go:64] FLAG: --feature-gates="" Sep 11 14:40:09.049122 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042698 2583 flags.go:64] FLAG: --file-check-frequency="20s" Sep 11 14:40:09.049122 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042716 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 11 14:40:09.049122 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042719 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 11 14:40:09.049122 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042722 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 11 14:40:09.049122 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042725 2583 flags.go:64] FLAG: --healthz-port="10248" Sep 11 14:40:09.049122 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042729 2583 flags.go:64] FLAG: --help="false" Sep 11 14:40:09.049122 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042732 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-135-39.ec2.internal" Sep 11 14:40:09.049122 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042735 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 11 14:40:09.049122 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042738 2583 flags.go:64] FLAG: --http-check-frequency="20s" Sep 11 14:40:09.049122 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042741 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 11 14:40:09.049122 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042745 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 11 14:40:09.049122 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042748 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 11 14:40:09.049122 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042752 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 11 14:40:09.049122 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042755 2583 flags.go:64] FLAG: --image-service-endpoint="" Sep 11 14:40:09.049122 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042758 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 11 14:40:09.049122 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042760 2583 flags.go:64] FLAG: --kube-api-burst="100" Sep 11 14:40:09.049122 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042764 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 11 14:40:09.049774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042767 2583 flags.go:64] FLAG: --kube-api-qps="50" Sep 11 14:40:09.049774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042770 2583 flags.go:64] FLAG: --kube-reserved="" Sep 11 14:40:09.049774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042773 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 11 14:40:09.049774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042776 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 11 14:40:09.049774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042779 2583 flags.go:64] FLAG: --kubelet-cgroups="" Sep 11 14:40:09.049774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042781 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 11 14:40:09.049774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042784 2583 flags.go:64] FLAG: --lock-file="" Sep 11 14:40:09.049774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042787 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 11 14:40:09.049774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042790 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 11 14:40:09.049774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042793 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 11 14:40:09.049774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042799 2583 flags.go:64] FLAG: --log-json-split-stream="false" Sep 11 14:40:09.049774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042802 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 11 14:40:09.049774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042805 2583 flags.go:64] FLAG: --log-text-split-stream="false" Sep 11 14:40:09.049774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042807 2583 flags.go:64] FLAG: --logging-format="text" Sep 11 14:40:09.049774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042810 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 11 14:40:09.049774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042813 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 11 14:40:09.049774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042816 2583 flags.go:64] FLAG: --manifest-url="" Sep 11 14:40:09.049774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042819 2583 flags.go:64] FLAG: --manifest-url-header="" Sep 11 14:40:09.049774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042823 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 11 14:40:09.049774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042826 2583 flags.go:64] FLAG: --max-open-files="1000000" Sep 11 14:40:09.049774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042831 2583 flags.go:64] FLAG: --max-pods="110" Sep 11 14:40:09.049774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042834 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 11 14:40:09.049774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042837 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 11 14:40:09.049774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042840 2583 flags.go:64] FLAG: --memory-manager-policy="None" Sep 11 14:40:09.049774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042843 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 11 14:40:09.050384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042846 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 11 14:40:09.050384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042848 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 11 14:40:09.050384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042852 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 11 14:40:09.050384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042859 2583 flags.go:64] FLAG: --node-status-max-images="50" Sep 11 14:40:09.050384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042862 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 11 14:40:09.050384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042865 2583 flags.go:64] FLAG: --oom-score-adj="-999" Sep 11 14:40:09.050384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042868 2583 flags.go:64] FLAG: --pod-cidr="" Sep 11 14:40:09.050384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042871 2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 11 14:40:09.050384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042876 2583 flags.go:64] FLAG: --pod-manifest-path="" Sep 11 14:40:09.050384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042879 2583 flags.go:64] FLAG: --pod-max-pids="-1" Sep 11 14:40:09.050384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042882 2583 flags.go:64] FLAG: --pods-per-core="0" Sep 11 14:40:09.050384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042885 2583 flags.go:64] FLAG: --port="10250" Sep 11 14:40:09.050384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042888 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 11 14:40:09.050384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042891 2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03702845408ae2eff" Sep 11 14:40:09.050384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042894 2583 flags.go:64] FLAG: --qos-reserved="" Sep 11 14:40:09.050384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042897 2583 flags.go:64] FLAG: --read-only-port="10255" Sep 11 14:40:09.050384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042900 2583 flags.go:64] FLAG: --register-node="true" Sep 11 14:40:09.050384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042903 2583 flags.go:64] FLAG: --register-with-taints="" Sep 11 14:40:09.050384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042906 2583 flags.go:64] FLAG: --registry-burst="10" Sep 11 14:40:09.050384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042909 2583 flags.go:64] FLAG: --registry-qps="5" Sep 11 14:40:09.050384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042912 2583 flags.go:64] FLAG: --reserved-cpus="" Sep 11 14:40:09.050384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042914 2583 flags.go:64] FLAG: --reserved-memory="" Sep 11 14:40:09.050384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042918 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 11 14:40:09.050384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042921 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 11 14:40:09.050384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042924 2583 flags.go:64] FLAG: --rotate-certificates="false" Sep 11 14:40:09.051003 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042927 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 11 14:40:09.051003 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042930 2583 flags.go:64] FLAG: --runonce="false" Sep 11 14:40:09.051003 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042934 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 11 14:40:09.051003 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042937 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 11 14:40:09.051003 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042940 2583 flags.go:64] FLAG: --seccomp-default="false" Sep 11 14:40:09.051003 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042944 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 11 14:40:09.051003 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042947 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 11 14:40:09.051003 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042950 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 11 14:40:09.051003 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042953 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 11 14:40:09.051003 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042956 2583 flags.go:64] FLAG: --storage-driver-password="root" Sep 11 14:40:09.051003 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042960 2583 flags.go:64] FLAG: --storage-driver-secure="false" Sep 11 14:40:09.051003 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042963 2583 flags.go:64] FLAG: --storage-driver-table="stats" Sep 11 14:40:09.051003 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042966 2583 flags.go:64] FLAG: --storage-driver-user="root" Sep 11 14:40:09.051003 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042968 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 11 14:40:09.051003 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042971 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 11 14:40:09.051003 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042974 2583 flags.go:64] FLAG: --system-cgroups="" Sep 11 14:40:09.051003 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042977 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 11 14:40:09.051003 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042983 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 11 14:40:09.051003 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042985 2583 flags.go:64] FLAG: --tls-cert-file="" Sep 11 14:40:09.051003 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042988 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 11 14:40:09.051003 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042992 2583 flags.go:64] FLAG: --tls-min-version="" Sep 11 14:40:09.051003 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042994 2583 flags.go:64] FLAG: --tls-private-key-file="" Sep 11 14:40:09.051003 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.042997 2583 flags.go:64] FLAG: --topology-manager-policy="none" Sep 11 14:40:09.051003 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.043000 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 11 14:40:09.051003 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.043003 2583 flags.go:64] FLAG: --topology-manager-scope="container" Sep 11 14:40:09.051599 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.043005 2583 flags.go:64] FLAG: --v="2" Sep 11 14:40:09.051599 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.043009 2583 flags.go:64] FLAG: --version="false" Sep 11 14:40:09.051599 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.043013 2583 flags.go:64] FLAG: --vmodule="" Sep 11 14:40:09.051599 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.043017 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 11 14:40:09.051599 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.043020 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 11 14:40:09.051599 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043130 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:40:09.051599 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043133 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:40:09.051599 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043136 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:40:09.051599 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043139 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:40:09.051599 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043144 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:40:09.051599 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043146 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:40:09.051599 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043149 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:40:09.051599 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043152 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:40:09.051599 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043154 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:40:09.051599 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043157 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:40:09.051599 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043160 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:40:09.051599 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043164 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:40:09.051599 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043168 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:40:09.051599 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043171 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:40:09.051599 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043173 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:40:09.051599 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043176 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:40:09.052158 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043179 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:40:09.052158 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043182 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:40:09.052158 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043185 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:40:09.052158 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043188 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:40:09.052158 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043191 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:40:09.052158 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043194 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:40:09.052158 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043197 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:40:09.052158 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043199 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:40:09.052158 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043202 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:40:09.052158 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043204 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:40:09.052158 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043207 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:40:09.052158 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043209 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:40:09.052158 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043213 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:40:09.052158 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043216 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:40:09.052158 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043219 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:40:09.052158 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043221 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:40:09.052158 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043224 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:40:09.052158 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043227 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:40:09.052158 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043229 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:40:09.052614 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043232 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:40:09.052614 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043236 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:40:09.052614 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043238 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:40:09.052614 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043241 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:40:09.052614 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043244 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:40:09.052614 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043246 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:40:09.052614 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043249 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:40:09.052614 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043251 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:40:09.052614 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043254 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:40:09.052614 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043257 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:40:09.052614 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043259 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:40:09.052614 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043262 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:40:09.052614 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043264 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:40:09.052614 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043267 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:40:09.052614 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043269 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:40:09.052614 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043272 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:40:09.052614 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043274 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:40:09.052614 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043277 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:40:09.052614 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043280 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:40:09.052614 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043282 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:40:09.053164 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043285 2583 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:40:09.053164 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043287 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:40:09.053164 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043290 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:40:09.053164 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043292 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:40:09.053164 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043295 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:40:09.053164 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043297 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:40:09.053164 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043300 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:40:09.053164 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043302 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:40:09.053164 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043305 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:40:09.053164 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043307 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:40:09.053164 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043310 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:40:09.053164 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043312 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:40:09.053164 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043314 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:40:09.053164 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043319 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:40:09.053164 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043321 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:40:09.053164 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043324 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:40:09.053164 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043326 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:40:09.053164 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043329 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:40:09.053164 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043331 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:40:09.053164 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043334 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:40:09.053673 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043336 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:40:09.053673 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043339 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:40:09.053673 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043341 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:40:09.053673 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043344 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:40:09.053673 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043346 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:40:09.053673 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043349 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:40:09.053673 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043351 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:40:09.053673 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043354 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:40:09.053673 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043356 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:40:09.053673 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043359 2583 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:40:09.053673 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043362 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:40:09.053673 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043364 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:40:09.053673 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043367 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:40:09.053673 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043369 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:40:09.053673 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043372 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:40:09.053673 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043374 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:40:09.053673 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043377 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:40:09.053673 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043379 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:40:09.053673 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043382 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:40:09.053673 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043384 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:40:09.054171 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043386 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:40:09.054171 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043390 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:40:09.054171 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043393 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:40:09.054171 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.043396 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:40:09.054171 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.043909 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 14:40:09.054171 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.050540 2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 11 14:40:09.054171 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.050557 2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 11 14:40:09.054171 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.051340 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 11 14:40:09.054171 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.051348 2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 11 14:40:09.054171 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051404 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:40:09.054171 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051409 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:40:09.054171 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051412 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:40:09.054171 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051416 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:40:09.054171 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051419 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:40:09.054534 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051422 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:40:09.054534 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051424 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:40:09.054534 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051427 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:40:09.054534 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051429 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:40:09.054534 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051432 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:40:09.054534 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051434 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:40:09.054534 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051437 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:40:09.054534 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051439 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:40:09.054534 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051442 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:40:09.054534 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051445 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:40:09.054534 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051448 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:40:09.054534 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051451 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:40:09.054534 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051453 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:40:09.054534 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051458 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:40:09.054534 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051462 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:40:09.054534 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051465 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:40:09.054534 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051468 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:40:09.054534 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051471 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:40:09.054534 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051474 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:40:09.054991 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051476 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:40:09.054991 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051479 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:40:09.054991 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051481 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:40:09.054991 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051484 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:40:09.054991 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051487 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:40:09.054991 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051490 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:40:09.054991 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051493 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:40:09.054991 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051496 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:40:09.054991 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051498 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:40:09.054991 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051501 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:40:09.054991 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051503 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:40:09.054991 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051506 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:40:09.054991 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051509 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:40:09.054991 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051511 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:40:09.054991 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051514 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:40:09.054991 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051516 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:40:09.054991 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051519 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:40:09.054991 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051521 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:40:09.054991 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051523 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:40:09.054991 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051525 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:40:09.055502 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051528 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:40:09.055502 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051531 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:40:09.055502 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051533 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:40:09.055502 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051536 2583 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:40:09.055502 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051538 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:40:09.055502 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051541 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:40:09.055502 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051544 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:40:09.055502 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051547 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:40:09.055502 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051549 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:40:09.055502 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051552 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:40:09.055502 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051554 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:40:09.055502 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051557 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:40:09.055502 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051560 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:40:09.055502 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051562 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:40:09.055502 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051565 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:40:09.055502 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051568 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:40:09.055502 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051571 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:40:09.055502 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051573 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:40:09.055502 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051576 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:40:09.055502 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051579 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:40:09.056037 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051581 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:40:09.056037 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051584 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:40:09.056037 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051586 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:40:09.056037 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051589 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:40:09.056037 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051591 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:40:09.056037 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051594 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:40:09.056037 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051596 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:40:09.056037 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051599 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:40:09.056037 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051601 2583 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:40:09.056037 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051604 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:40:09.056037 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051607 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:40:09.056037 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051609 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:40:09.056037 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051612 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:40:09.056037 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051614 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:40:09.056037 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051617 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:40:09.056037 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051620 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:40:09.056037 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051622 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:40:09.056037 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051625 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:40:09.056037 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051627 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:40:09.056478 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051631 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:40:09.056478 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051635 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:40:09.056478 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051638 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:40:09.056478 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051641 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:40:09.056478 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051643 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:40:09.056478 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051646 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:40:09.056478 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051649 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:40:09.056478 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051651 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:40:09.056478 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051654 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:40:09.056478 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051656 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:40:09.056478 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051659 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:40:09.056478 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051661 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:40:09.056478 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051664 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:40:09.056478 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051667 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:40:09.056478 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051669 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:40:09.056478 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051673 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:40:09.056870 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.051678 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 14:40:09.056870 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051800 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:40:09.056870 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051806 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:40:09.056870 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051809 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:40:09.056870 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051812 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:40:09.056870 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051815 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:40:09.056870 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051818 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:40:09.056870 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051821 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:40:09.056870 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051824 2583 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:40:09.056870 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051826 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:40:09.056870 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051829 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:40:09.056870 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051831 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:40:09.056870 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051835 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:40:09.056870 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051837 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:40:09.056870 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051840 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:40:09.057259 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051843 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:40:09.057259 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051845 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:40:09.057259 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051848 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:40:09.057259 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051850 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:40:09.057259 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051853 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:40:09.057259 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051856 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:40:09.057259 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051858 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:40:09.057259 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051861 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:40:09.057259 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051863 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:40:09.057259 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051866 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:40:09.057259 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051868 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:40:09.057259 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051871 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:40:09.057259 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051874 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:40:09.057259 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051876 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:40:09.057259 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051879 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:40:09.057259 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051881 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:40:09.057259 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051884 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:40:09.057259 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051886 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:40:09.057259 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051889 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:40:09.057259 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051891 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:40:09.057764 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051894 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:40:09.057764 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051896 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:40:09.057764 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051898 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:40:09.057764 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051901 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:40:09.057764 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051904 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:40:09.057764 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051906 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:40:09.057764 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051908 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:40:09.057764 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051911 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:40:09.057764 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051913 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:40:09.057764 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051916 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:40:09.057764 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051920 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:40:09.057764 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051923 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:40:09.057764 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051926 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:40:09.057764 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051929 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:40:09.057764 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051932 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:40:09.057764 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051935 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:40:09.057764 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051938 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:40:09.057764 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051940 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:40:09.057764 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051942 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:40:09.058241 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051945 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:40:09.058241 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051948 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:40:09.058241 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051950 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:40:09.058241 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051953 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:40:09.058241 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051955 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:40:09.058241 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051957 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:40:09.058241 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051960 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:40:09.058241 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051962 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:40:09.058241 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051965 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:40:09.058241 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051968 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:40:09.058241 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051970 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:40:09.058241 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051973 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:40:09.058241 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051976 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:40:09.058241 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051978 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:40:09.058241 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051980 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:40:09.058241 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051983 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:40:09.058241 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051985 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:40:09.058241 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051988 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:40:09.058241 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051991 2583 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:40:09.058241 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051994 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:40:09.058721 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.051997 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:40:09.058721 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.052000 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:40:09.058721 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.052004 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:40:09.058721 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.052007 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:40:09.058721 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.052010 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:40:09.058721 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.052013 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:40:09.058721 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.052016 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:40:09.058721 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.052018 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:40:09.058721 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.052021 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:40:09.058721 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.052023 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:40:09.058721 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.052026 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:40:09.058721 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.052028 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:40:09.058721 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.052031 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:40:09.058721 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.052034 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:40:09.058721 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.052036 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:40:09.058721 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.052039 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:40:09.058721 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.052041 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:40:09.058721 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.052043 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:40:09.058721 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.052046 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:40:09.059169 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.052048 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:40:09.059169 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.052051 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:40:09.059169 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.052054 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:40:09.059169 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.052056 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:40:09.059169 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.052058 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:40:09.059169 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.052061 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:40:09.059169 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:09.052063 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:40:09.059169 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.052068 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 14:40:09.059169 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.052173 2583 server.go:962] "Client rotation is on, will bootstrap in background" Sep 11 14:40:09.059169 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.054077 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 11 14:40:09.059169 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.055442 2583 server.go:1019] "Starting client certificate rotation" Sep 11 14:40:09.059169 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.055537 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 14:40:09.059169 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.055566 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 14:40:09.059476 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.057784 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 14:40:09.061041 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.061014 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 14:40:09.071946 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.071924 2583 log.go:25] "Validated CRI v1 runtime API" Sep 11 14:40:09.075202 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.075183 2583 log.go:25] "Validated CRI v1 image API" Sep 11 14:40:09.076018 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.076001 2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 11 14:40:09.080437 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.080418 2583 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8b6465d2-d94f-4660-b42d-806ec9d379ce:/dev/nvme0n1p3 b96b321f-dafc-47c2-a7c3-96b463088242:/dev/nvme0n1p4] Sep 11 14:40:09.080505 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.080437 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 11 14:40:09.086544 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.086431 2583 manager.go:216] Machine: {Timestamp:2026-09-11 14:40:09.084065974 +0000 UTC m=+0.328051622 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3198373 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec243bb5ae0f0a3ec4f8ca0fb7e9804f SystemUUID:ec243bb5-ae0f-0a3e-c4f8-ca0fb7e9804f BootID:acbd07d7-4d4b-4bbe-99da-5a3e4f53a0b2 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:29:f0:41:91:cd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:29:f0:41:91:cd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:56:08:e3:f3:d7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 11 14:40:09.086544 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.086536 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 11 14:40:09.086677 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.086615 2583 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 11 14:40:09.086866 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.086851 2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 14:40:09.089534 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.089505 2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 11 14:40:09.089675 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.089538 2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-39.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 11 14:40:09.090157 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.090148 2583 topology_manager.go:138] "Creating topology manager with none policy" Sep 11 14:40:09.090190 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.090159 2583 container_manager_linux.go:309] "Creating device plugin manager" Sep 11 14:40:09.090190 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.090167 2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 14:40:09.090799 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.090789 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 14:40:09.091479 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.091470 2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 11 14:40:09.093819 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.093807 2583 state_mem.go:36] "Initialized new in-memory state store" Sep 11 14:40:09.093932 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.093921 2583 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 11 14:40:09.093971 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.093955 2583 kubelet.go:486] "Attempting to sync node with API server" Sep 11 14:40:09.093971 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.093967 2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 11 14:40:09.094032 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.093981 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 11 14:40:09.094032 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.093993 2583 kubelet.go:398] "Adding apiserver pod source" Sep 11 14:40:09.094032 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.094000 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 11 14:40:09.094448 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.094420 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 14:40:09.094540 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.094459 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 14:40:09.096601 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.096582 2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 11 14:40:09.098626 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.098609 2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 11 14:40:09.098675 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.098642 2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 11 14:40:09.100117 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.100105 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 11 14:40:09.100152 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.100122 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 11 14:40:09.100152 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.100129 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 11 14:40:09.100152 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.100133 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 11 14:40:09.100152 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.100146 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 11 14:40:09.100258 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.100154 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 11 14:40:09.100258 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.100170 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 11 14:40:09.100258 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.100175 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 11 14:40:09.100258 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.100185 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 11 14:40:09.100258 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.100190 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 11 14:40:09.100258 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.100198 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 11 14:40:09.100258 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.100203 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 11 14:40:09.100777 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.100766 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 11 14:40:09.100777 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.100777 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 11 14:40:09.104862 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.104844 2583 server.go:1268] "Started kubelet" Sep 11 14:40:09.105104 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.105031 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 11 14:40:09.105104 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.104960 2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 11 14:40:09.105273 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.105123 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 11 14:40:09.106034 ip-10-0-135-39 systemd[1]: Started Kubernetes Kubelet. Sep 11 14:40:09.106528 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.106323 2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 11 14:40:09.106939 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.106914 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-39.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 11 14:40:09.107025 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:09.106939 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 11 14:40:09.107301 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:09.107279 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-39.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 11 14:40:09.108175 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.108075 2583 server.go:310] "Adding debug handlers to kubelet server" Sep 11 14:40:09.111763 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:09.111741 2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 11 14:40:09.113438 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.113418 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 11 14:40:09.113855 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.113839 2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 11 14:40:09.114003 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:09.113986 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-39.ec2.internal\" not found" Sep 11 14:40:09.114364 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.114344 2583 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 11 14:40:09.114443 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.114368 2583 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 11 14:40:09.114443 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.114344 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 11 14:40:09.114539 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.114456 2583 reconstruct.go:97] "Volume reconstruction finished" Sep 11 14:40:09.114539 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.114465 2583 reconciler.go:29] "Reconciler: start to sync state" Sep 11 14:40:09.114773 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.114758 2583 factory.go:55] Registering systemd factory Sep 11 14:40:09.114773 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.114775 2583 factory.go:223] Registration of the systemd container factory successfully Sep 11 14:40:09.114980 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.114968 2583 factory.go:153] Registering CRI-O factory Sep 11 14:40:09.115034 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.114984 2583 factory.go:223] Registration of the crio container factory successfully Sep 11 14:40:09.115094 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.115044 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 11 14:40:09.115094 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.115067 2583 factory.go:103] Registering Raw factory Sep 11 14:40:09.115094 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.115082 2583 manager.go:1196] Started watching for new ooms in manager Sep 11 14:40:09.115570 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:09.114627 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-39.ec2.internal.18d44b2d1cfbdfad default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-39.ec2.internal,UID:ip-10-0-135-39.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-39.ec2.internal,},FirstTimestamp:2026-09-11 14:40:09.104809901 +0000 UTC m=+0.348795555,LastTimestamp:2026-09-11 14:40:09.104809901 +0000 UTC m=+0.348795555,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-39.ec2.internal,}" Sep 11 14:40:09.115822 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.115810 2583 manager.go:349] Starting recovery of all containers Sep 11 14:40:09.118628 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.118607 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m4jfl" Sep 11 14:40:09.125125 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:09.125090 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 11 14:40:09.125236 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:09.125145 2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-39.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 11 14:40:09.126728 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.126697 2583 manager.go:354] Recovery completed Sep 11 14:40:09.128180 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.128163 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m4jfl" Sep 11 14:40:09.131683 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.131668 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:40:09.134489 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.134467 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-39.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:40:09.134570 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.134510 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-39.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:40:09.134570 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.134526 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-39.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:40:09.135663 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.135644 2583 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 11 14:40:09.135663 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.135663 2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 11 14:40:09.135811 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.135681 2583 state_mem.go:36] "Initialized new in-memory state store" Sep 11 14:40:09.135811 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.135744 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:40:09.136621 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.136601 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-39.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:40:09.136700 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.136633 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-39.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:40:09.136700 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.136655 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-39.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:40:09.136824 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.136772 2583 policy_none.go:49] "None policy: Start" Sep 11 14:40:09.136824 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.136788 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 11 14:40:09.136908 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.136831 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 11 14:40:09.138099 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.138086 2583 policy_none.go:47] "Start" Sep 11 14:40:09.161383 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.161369 2583 manager.go:337] "Starting Device Plugin manager" Sep 11 14:40:09.175296 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:09.161411 2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 11 14:40:09.175296 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.161425 2583 server.go:87] "Starting device plugin registration server" Sep 11 14:40:09.175296 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.161752 2583 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 11 14:40:09.175296 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.161766 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 11 14:40:09.175296 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.161956 2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 11 14:40:09.175296 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.162091 2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 11 14:40:09.175296 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.162101 2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 11 14:40:09.175296 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:09.162774 2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 11 14:40:09.175296 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:09.162826 2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-39.ec2.internal\" not found" Sep 11 14:40:09.262730 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.262687 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:40:09.263763 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.263742 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-39.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:40:09.263866 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.263777 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-39.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:40:09.263866 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.263791 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-39.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:40:09.263866 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.263827 2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-39.ec2.internal" Sep 11 14:40:09.272981 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.272964 2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-39.ec2.internal" Sep 11 14:40:09.273079 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:09.272990 2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-39.ec2.internal\": node \"ip-10-0-135-39.ec2.internal\" not found" Sep 11 14:40:09.289213 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:09.289109 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-39.ec2.internal\" not found" Sep 11 14:40:09.322670 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.322635 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 11 14:40:09.323777 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.323757 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 11 14:40:09.323861 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.323814 2583 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 11 14:40:09.323861 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.323843 2583 kubelet.go:2448] "Starting kubelet main sync loop" Sep 11 14:40:09.323952 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:09.323887 2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 11 14:40:09.326558 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.326540 2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:40:09.389382 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:09.389363 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-39.ec2.internal\" not found" Sep 11 14:40:09.424727 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.424685 2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-39.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-39.ec2.internal"] Sep 11 14:40:09.424791 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.424787 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:40:09.425694 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.425672 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-39.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:40:09.425773 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.425720 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-39.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:40:09.425773 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.425735 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-39.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:40:09.426939 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.426927 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:40:09.427067 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.427054 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-39.ec2.internal" Sep 11 14:40:09.427110 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.427081 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:40:09.427654 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.427640 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-39.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:40:09.427654 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.427651 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-39.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:40:09.427770 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.427667 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-39.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:40:09.427770 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.427670 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-39.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:40:09.427770 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.427716 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-39.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:40:09.427865 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.427679 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-39.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:40:09.428842 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.428828 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-39.ec2.internal" Sep 11 14:40:09.428888 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.428854 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:40:09.429524 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.429507 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-39.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:40:09.429594 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.429534 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-39.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:40:09.429594 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.429547 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-39.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:40:09.456218 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:09.456200 2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-39.ec2.internal\" not found" node="ip-10-0-135-39.ec2.internal" Sep 11 14:40:09.460471 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:09.460455 2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-39.ec2.internal\" not found" node="ip-10-0-135-39.ec2.internal" Sep 11 14:40:09.489759 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:09.489739 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-39.ec2.internal\" not found" Sep 11 14:40:09.516151 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.516129 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/34be09161f9ee3edbfcdbbe22b0b8213-config\") pod \"kube-apiserver-proxy-ip-10-0-135-39.ec2.internal\" (UID: \"34be09161f9ee3edbfcdbbe22b0b8213\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-39.ec2.internal" Sep 11 14:40:09.516216 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.516156 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a6e4f2dac72fee1f4d30b95dd4ee38cb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-39.ec2.internal\" (UID: \"a6e4f2dac72fee1f4d30b95dd4ee38cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-39.ec2.internal" Sep 11 14:40:09.516216 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.516171 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6e4f2dac72fee1f4d30b95dd4ee38cb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-39.ec2.internal\" (UID: \"a6e4f2dac72fee1f4d30b95dd4ee38cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-39.ec2.internal" Sep 11 14:40:09.590676 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:09.590627 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-39.ec2.internal\" not found" Sep 11 14:40:09.617052 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.617029 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/34be09161f9ee3edbfcdbbe22b0b8213-config\") pod \"kube-apiserver-proxy-ip-10-0-135-39.ec2.internal\" (UID: \"34be09161f9ee3edbfcdbbe22b0b8213\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-39.ec2.internal" Sep 11 14:40:09.617110 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.617066 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a6e4f2dac72fee1f4d30b95dd4ee38cb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-39.ec2.internal\" (UID: \"a6e4f2dac72fee1f4d30b95dd4ee38cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-39.ec2.internal" Sep 11 14:40:09.617110 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.617085 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/34be09161f9ee3edbfcdbbe22b0b8213-config\") pod \"kube-apiserver-proxy-ip-10-0-135-39.ec2.internal\" (UID: \"34be09161f9ee3edbfcdbbe22b0b8213\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-39.ec2.internal" Sep 11 14:40:09.617110 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.617091 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6e4f2dac72fee1f4d30b95dd4ee38cb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-39.ec2.internal\" (UID: \"a6e4f2dac72fee1f4d30b95dd4ee38cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-39.ec2.internal" Sep 11 14:40:09.617234 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.617128 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6e4f2dac72fee1f4d30b95dd4ee38cb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-39.ec2.internal\" (UID: \"a6e4f2dac72fee1f4d30b95dd4ee38cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-39.ec2.internal" Sep 11 14:40:09.617234 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.617147 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a6e4f2dac72fee1f4d30b95dd4ee38cb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-39.ec2.internal\" (UID: \"a6e4f2dac72fee1f4d30b95dd4ee38cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-39.ec2.internal" Sep 11 14:40:09.691164 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:09.691144 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-39.ec2.internal\" not found" Sep 11 14:40:09.758684 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.758661 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-39.ec2.internal" Sep 11 14:40:09.763295 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:09.763279 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-39.ec2.internal" Sep 11 14:40:09.791437 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:09.791417 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-39.ec2.internal\" not found" Sep 11 14:40:09.891952 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:09.891890 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-39.ec2.internal\" not found" Sep 11 14:40:09.993563 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:09.993534 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-39.ec2.internal\" not found" Sep 11 14:40:10.055300 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:10.055274 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 11 14:40:10.055726 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:10.055416 2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 14:40:10.095935 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:10.095910 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-39.ec2.internal\" not found" Sep 11 14:40:10.132275 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:10.132244 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 14:35:09 +0000 UTC" deadline="2028-04-27 09:26:57.249266711 +0000 UTC" Sep 11 14:40:10.132275 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:10.132269 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14250h46m47.117000868s" Sep 11 14:40:10.198296 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:10.198241 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-39.ec2.internal\" not found" Sep 11 14:40:10.297429 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:10.297403 2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:40:10.298465 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:10.298444 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-39.ec2.internal\" not found" Sep 11 14:40:10.322479 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:10.322440 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34be09161f9ee3edbfcdbbe22b0b8213.slice/crio-6e980addd7ccd7dccecea5c579568758a6bb4c909b148cc5900955066ec0bd45 WatchSource:0}: Error finding container 6e980addd7ccd7dccecea5c579568758a6bb4c909b148cc5900955066ec0bd45: Status 404 returned error can't find the container with id 6e980addd7ccd7dccecea5c579568758a6bb4c909b148cc5900955066ec0bd45 Sep 11 14:40:10.322706 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:10.322688 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e4f2dac72fee1f4d30b95dd4ee38cb.slice/crio-d0a39d8e951fd38859e0504990b81a54130b6d96b501a0f4e2b076cc8a9cab40 WatchSource:0}: Error finding container d0a39d8e951fd38859e0504990b81a54130b6d96b501a0f4e2b076cc8a9cab40: Status 404 returned error can't find the container with id d0a39d8e951fd38859e0504990b81a54130b6d96b501a0f4e2b076cc8a9cab40 Sep 11 14:40:10.327134 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:10.327119 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:40:10.327369 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:10.327308 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-39.ec2.internal" event={"ID":"34be09161f9ee3edbfcdbbe22b0b8213","Type":"ContainerStarted","Data":"6e980addd7ccd7dccecea5c579568758a6bb4c909b148cc5900955066ec0bd45"} Sep 11 14:40:10.328585 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:10.328563 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-39.ec2.internal" event={"ID":"a6e4f2dac72fee1f4d30b95dd4ee38cb","Type":"ContainerStarted","Data":"d0a39d8e951fd38859e0504990b81a54130b6d96b501a0f4e2b076cc8a9cab40"} Sep 11 14:40:10.399203 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:10.399178 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-39.ec2.internal\" not found" Sep 11 14:40:10.501022 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:10.500971 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-39.ec2.internal\" not found" Sep 11 14:40:10.509330 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:10.509306 2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:40:10.526905 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:10.526886 2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:40:10.614560 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:10.614534 2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-39.ec2.internal" Sep 11 14:40:10.629513 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:10.629494 2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 14:40:10.630560 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:10.630545 2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-39.ec2.internal" Sep 11 14:40:10.642168 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:10.642142 2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 14:40:11.096612 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.096571 2583 apiserver.go:52] "Watching apiserver" Sep 11 14:40:11.105217 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.105191 2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 11 14:40:11.107136 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.107102 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-mlrnk","kube-system/konnectivity-agent-vfmp4","openshift-dns/node-resolver-ppnhx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-39.ec2.internal","openshift-multus/multus-97b8j","openshift-multus/multus-additional-cni-plugins-75zzz","openshift-multus/network-metrics-daemon-brx7q","openshift-network-diagnostics/network-check-target-5hp9w","kube-system/kube-apiserver-proxy-ip-10-0-135-39.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz","openshift-cluster-node-tuning-operator/tuned-7mbt2","openshift-image-registry/node-ca-4vlml","openshift-network-operator/iptables-alerter-m7tfj","openshift-ovn-kubernetes/ovnkube-node-vlg7j"] Sep 11 14:40:11.108788 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.108762 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" Sep 11 14:40:11.110307 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.110287 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m7tfj" Sep 11 14:40:11.111547 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.111284 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ppnhx" Sep 11 14:40:11.112647 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.112628 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 11 14:40:11.112891 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.112869 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 11 14:40:11.113171 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.113152 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gdc62\"" Sep 11 14:40:11.113351 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.113334 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 11 14:40:11.113749 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.113521 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 11 14:40:11.113749 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.113569 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 11 14:40:11.113749 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.113595 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 11 14:40:11.113749 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.113685 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 11 14:40:11.113749 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.113706 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4vbsc\"" Sep 11 14:40:11.114088 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.113874 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.114626 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.114569 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.116970 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.115864 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 11 14:40:11.116970 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.115976 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-75zzz" Sep 11 14:40:11.116970 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.116168 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 11 14:40:11.116970 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.116365 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-29mf9\"" Sep 11 14:40:11.116970 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.116648 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 11 14:40:11.116970 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.116815 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ddcg5\"" Sep 11 14:40:11.116970 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.116901 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 11 14:40:11.116970 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.116935 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 11 14:40:11.117456 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.117003 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 11 14:40:11.117456 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.117270 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 11 14:40:11.122355 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.120390 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 11 14:40:11.122355 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.120557 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 11 14:40:11.122355 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.120708 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cwmpv\"" Sep 11 14:40:11.122355 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.120962 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 11 14:40:11.122355 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.121158 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 11 14:40:11.122355 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.121248 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:40:11.122355 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:11.121333 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hp9w" podUID="ecdbcfa3-6309-46be-a9c9-ea8eb658de34" Sep 11 14:40:11.122355 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.121509 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 11 14:40:11.122355 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.121747 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-464jn\"" Sep 11 14:40:11.125557 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.123364 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:11.125557 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:11.123454 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brx7q" podUID="73182ce3-cb4b-4fcb-992e-85d116cf7e15" Sep 11 14:40:11.125557 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.124501 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-host-kubelet\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.125557 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.124544 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0b0394a-1326-4465-b674-23c0c6fbe017-cnibin\") pod \"multus-additional-cni-plugins-75zzz\" (UID: \"c0b0394a-1326-4465-b674-23c0c6fbe017\") " pod="openshift-multus/multus-additional-cni-plugins-75zzz" Sep 11 14:40:11.125557 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.124575 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-multus-cni-dir\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.125557 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.124607 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b82ed19-be9c-4a91-bc2d-b1ff936194da-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h7fcz\" (UID: \"7b82ed19-be9c-4a91-bc2d-b1ff936194da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" Sep 11 14:40:11.125557 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.124640 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-run-openvswitch\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.125557 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.124676 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/40743b05-b892-4d5a-bd13-46fb3822ad56-ovnkube-config\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.125557 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.124706 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/40743b05-b892-4d5a-bd13-46fb3822ad56-env-overrides\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.125557 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.124764 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0b0394a-1326-4465-b674-23c0c6fbe017-cni-binary-copy\") pod \"multus-additional-cni-plugins-75zzz\" (UID: \"c0b0394a-1326-4465-b674-23c0c6fbe017\") " pod="openshift-multus/multus-additional-cni-plugins-75zzz" Sep 11 14:40:11.125557 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.124824 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0b0394a-1326-4465-b674-23c0c6fbe017-tuning-conf-dir\") pod \"multus-additional-cni-plugins-75zzz\" (UID: \"c0b0394a-1326-4465-b674-23c0c6fbe017\") " pod="openshift-multus/multus-additional-cni-plugins-75zzz" Sep 11 14:40:11.125557 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.124875 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-host-run-netns\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.125557 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.124912 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-etc-openvswitch\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.125557 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.124974 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-hostroot\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.125557 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.125026 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-host-run-multus-certs\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.125557 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.125058 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-host-slash\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.125557 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.125092 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-node-log\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.129432 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.125140 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/40743b05-b892-4d5a-bd13-46fb3822ad56-ovn-node-metrics-cert\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.129432 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.125178 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c0b0394a-1326-4465-b674-23c0c6fbe017-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-75zzz\" (UID: \"c0b0394a-1326-4465-b674-23c0c6fbe017\") " pod="openshift-multus/multus-additional-cni-plugins-75zzz" Sep 11 14:40:11.129432 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.125201 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 11 14:40:11.129432 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.125228 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d39f63c1-32f1-4c2a-8585-a3defd284630-cni-binary-copy\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.129432 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.125267 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa59047c-1d81-44d0-b3a3-9e56bf53511c-tmp-dir\") pod \"node-resolver-ppnhx\" (UID: \"fa59047c-1d81-44d0-b3a3-9e56bf53511c\") " pod="openshift-dns/node-resolver-ppnhx" Sep 11 14:40:11.129432 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.125301 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-host-var-lib-cni-multus\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.129432 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.125334 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-host-run-netns\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.129432 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.125398 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-system-cni-dir\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.129432 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.125430 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-cnibin\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.129432 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.125462 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-host-var-lib-cni-bin\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.129432 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.125495 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx4t5\" (UniqueName: \"kubernetes.io/projected/7b82ed19-be9c-4a91-bc2d-b1ff936194da-kube-api-access-lx4t5\") pod \"aws-ebs-csi-driver-node-h7fcz\" (UID: \"7b82ed19-be9c-4a91-bc2d-b1ff936194da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" Sep 11 14:40:11.129432 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.125548 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-var-lib-openvswitch\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.129432 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.125557 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 11 14:40:11.129432 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.125582 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8g5r\" (UniqueName: \"kubernetes.io/projected/ecdbcfa3-6309-46be-a9c9-ea8eb658de34-kube-api-access-p8g5r\") pod \"network-check-target-5hp9w\" (UID: \"ecdbcfa3-6309-46be-a9c9-ea8eb658de34\") " pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:40:11.129432 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.125614 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-multus-conf-dir\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.129432 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.125648 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-867wm\" (UniqueName: \"kubernetes.io/projected/40743b05-b892-4d5a-bd13-46fb3822ad56-kube-api-access-867wm\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.129432 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.125713 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f1b03dc4-474b-44b4-ba75-5feb9a72ebd3-iptables-alerter-script\") pod \"iptables-alerter-m7tfj\" (UID: \"f1b03dc4-474b-44b4-ba75-5feb9a72ebd3\") " pod="openshift-network-operator/iptables-alerter-m7tfj" Sep 11 14:40:11.130421 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.125744 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-host-run-k8s-cni-cncf-io\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.130421 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.125781 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-etc-kubernetes\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.130421 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.125832 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-host-var-lib-kubelet\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.130421 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.125867 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.130421 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.125902 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b82ed19-be9c-4a91-bc2d-b1ff936194da-sys-fs\") pod \"aws-ebs-csi-driver-node-h7fcz\" (UID: \"7b82ed19-be9c-4a91-bc2d-b1ff936194da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" Sep 11 14:40:11.130421 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.125932 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fa59047c-1d81-44d0-b3a3-9e56bf53511c-hosts-file\") pod \"node-resolver-ppnhx\" (UID: \"fa59047c-1d81-44d0-b3a3-9e56bf53511c\") " pod="openshift-dns/node-resolver-ppnhx" Sep 11 14:40:11.130421 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.125961 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-host-cni-netd\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.130421 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.126019 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0b0394a-1326-4465-b674-23c0c6fbe017-system-cni-dir\") pod \"multus-additional-cni-plugins-75zzz\" (UID: \"c0b0394a-1326-4465-b674-23c0c6fbe017\") " pod="openshift-multus/multus-additional-cni-plugins-75zzz" Sep 11 14:40:11.130421 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.126047 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b82ed19-be9c-4a91-bc2d-b1ff936194da-device-dir\") pod \"aws-ebs-csi-driver-node-h7fcz\" (UID: \"7b82ed19-be9c-4a91-bc2d-b1ff936194da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" Sep 11 14:40:11.130421 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.126079 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-systemd-units\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.130421 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.126156 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-host-cni-bin\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.130421 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.126221 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-run-systemd\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.130421 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.126440 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-host-run-ovn-kubernetes\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.130421 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.126513 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ztsw\" (UniqueName: \"kubernetes.io/projected/c0b0394a-1326-4465-b674-23c0c6fbe017-kube-api-access-6ztsw\") pod \"multus-additional-cni-plugins-75zzz\" (UID: \"c0b0394a-1326-4465-b674-23c0c6fbe017\") " pod="openshift-multus/multus-additional-cni-plugins-75zzz" Sep 11 14:40:11.130421 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.126552 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1b03dc4-474b-44b4-ba75-5feb9a72ebd3-host-slash\") pod \"iptables-alerter-m7tfj\" (UID: \"f1b03dc4-474b-44b4-ba75-5feb9a72ebd3\") " pod="openshift-network-operator/iptables-alerter-m7tfj" Sep 11 14:40:11.130421 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.126729 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:11.130421 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:11.126840 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlrnk" podUID="a1b523f4-d191-49bb-8e0e-4999e6fa1a92" Sep 11 14:40:11.131420 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.127705 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vfmp4" Sep 11 14:40:11.131420 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.128042 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9bkd\" (UniqueName: \"kubernetes.io/projected/d39f63c1-32f1-4c2a-8585-a3defd284630-kube-api-access-g9bkd\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.131420 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.128087 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-log-socket\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.131420 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.128125 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c0b0394a-1326-4465-b674-23c0c6fbe017-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-75zzz\" (UID: \"c0b0394a-1326-4465-b674-23c0c6fbe017\") " pod="openshift-multus/multus-additional-cni-plugins-75zzz" Sep 11 14:40:11.131420 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.128380 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-multus-socket-dir-parent\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.131420 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.128422 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/40743b05-b892-4d5a-bd13-46fb3822ad56-ovnkube-script-lib\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.131420 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.128453 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b82ed19-be9c-4a91-bc2d-b1ff936194da-registration-dir\") pod \"aws-ebs-csi-driver-node-h7fcz\" (UID: \"7b82ed19-be9c-4a91-bc2d-b1ff936194da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" Sep 11 14:40:11.131420 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.128532 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b82ed19-be9c-4a91-bc2d-b1ff936194da-socket-dir\") pod \"aws-ebs-csi-driver-node-h7fcz\" (UID: \"7b82ed19-be9c-4a91-bc2d-b1ff936194da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" Sep 11 14:40:11.131420 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.128547 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.131420 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.128652 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b82ed19-be9c-4a91-bc2d-b1ff936194da-etc-selinux\") pod \"aws-ebs-csi-driver-node-h7fcz\" (UID: \"7b82ed19-be9c-4a91-bc2d-b1ff936194da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" Sep 11 14:40:11.131420 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.128693 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzxlv\" (UniqueName: \"kubernetes.io/projected/fa59047c-1d81-44d0-b3a3-9e56bf53511c-kube-api-access-tzxlv\") pod \"node-resolver-ppnhx\" (UID: \"fa59047c-1d81-44d0-b3a3-9e56bf53511c\") " pod="openshift-dns/node-resolver-ppnhx" Sep 11 14:40:11.131420 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.128793 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-run-ovn\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.131420 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.128827 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mvxl\" (UniqueName: \"kubernetes.io/projected/f1b03dc4-474b-44b4-ba75-5feb9a72ebd3-kube-api-access-2mvxl\") pod \"iptables-alerter-m7tfj\" (UID: \"f1b03dc4-474b-44b4-ba75-5feb9a72ebd3\") " pod="openshift-network-operator/iptables-alerter-m7tfj" Sep 11 14:40:11.131420 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.128856 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d39f63c1-32f1-4c2a-8585-a3defd284630-multus-daemon-config\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.131420 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.130493 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 11 14:40:11.131420 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.130551 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dn9nk\"" Sep 11 14:40:11.131420 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.130712 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 11 14:40:11.131420 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.130932 2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 14:40:11.133888 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.132067 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 11 14:40:11.133888 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.132330 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wmmh4\"" Sep 11 14:40:11.133888 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.132396 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4vlml" Sep 11 14:40:11.133888 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.133175 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 11 14:40:11.135094 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.134690 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 11 14:40:11.135796 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.135360 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 11 14:40:11.135796 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.135589 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 11 14:40:11.136208 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.136008 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q9nd6\"" Sep 11 14:40:11.161935 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.161908 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hq2hj" Sep 11 14:40:11.172249 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.172014 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hq2hj" Sep 11 14:40:11.215918 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.215849 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 11 14:40:11.229039 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.229018 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-original-pull-secret\") pod \"global-pull-secret-syncer-mlrnk\" (UID: \"a1b523f4-d191-49bb-8e0e-4999e6fa1a92\") " pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:11.229154 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.229044 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-run\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.229154 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.229067 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-host-slash\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.229154 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.229084 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-node-log\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.229154 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.229125 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/40743b05-b892-4d5a-bd13-46fb3822ad56-ovn-node-metrics-cert\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.229381 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.229154 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-host-slash\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.229381 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.229159 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c0b0394a-1326-4465-b674-23c0c6fbe017-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-75zzz\" (UID: \"c0b0394a-1326-4465-b674-23c0c6fbe017\") " pod="openshift-multus/multus-additional-cni-plugins-75zzz" Sep 11 14:40:11.229381 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.229197 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d39f63c1-32f1-4c2a-8585-a3defd284630-cni-binary-copy\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.229381 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.229239 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-node-log\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.229381 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.229289 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7prq\" (UniqueName: \"kubernetes.io/projected/73182ce3-cb4b-4fcb-992e-85d116cf7e15-kube-api-access-q7prq\") pod \"network-metrics-daemon-brx7q\" (UID: \"73182ce3-cb4b-4fcb-992e-85d116cf7e15\") " pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:11.229381 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.229370 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8dc4e60b-0baa-45d8-93b2-63436c43e3be-host\") pod \"node-ca-4vlml\" (UID: \"8dc4e60b-0baa-45d8-93b2-63436c43e3be\") " pod="openshift-image-registry/node-ca-4vlml" Sep 11 14:40:11.229680 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.229403 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8dc4e60b-0baa-45d8-93b2-63436c43e3be-serviceca\") pod \"node-ca-4vlml\" (UID: \"8dc4e60b-0baa-45d8-93b2-63436c43e3be\") " pod="openshift-image-registry/node-ca-4vlml" Sep 11 14:40:11.229680 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.229431 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa59047c-1d81-44d0-b3a3-9e56bf53511c-tmp-dir\") pod \"node-resolver-ppnhx\" (UID: \"fa59047c-1d81-44d0-b3a3-9e56bf53511c\") " pod="openshift-dns/node-resolver-ppnhx" Sep 11 14:40:11.229680 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.229455 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-host-var-lib-cni-multus\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.229680 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.229514 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-675dv\" (UniqueName: \"kubernetes.io/projected/8dc4e60b-0baa-45d8-93b2-63436c43e3be-kube-api-access-675dv\") pod \"node-ca-4vlml\" (UID: \"8dc4e60b-0baa-45d8-93b2-63436c43e3be\") " pod="openshift-image-registry/node-ca-4vlml" Sep 11 14:40:11.229680 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.229551 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-etc-sysconfig\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.229680 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.229581 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-lib-modules\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.229680 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.229599 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-host-var-lib-cni-multus\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.229680 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.229633 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/804431e5-f91c-4e79-911e-6dcadceeb6ef-etc-tuned\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.230027 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.229824 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-host-run-netns\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.230027 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.229895 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-host-run-netns\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.230027 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.229901 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-system-cni-dir\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.230027 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.229957 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-cnibin\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.230027 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.229968 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-system-cni-dir\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.230027 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.229996 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-host-var-lib-cni-bin\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.230027 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230022 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-cnibin\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.230324 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230050 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-var-lib-kubelet\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.230324 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230146 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-host-var-lib-cni-bin\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.230324 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230148 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx4t5\" (UniqueName: \"kubernetes.io/projected/7b82ed19-be9c-4a91-bc2d-b1ff936194da-kube-api-access-lx4t5\") pod \"aws-ebs-csi-driver-node-h7fcz\" (UID: \"7b82ed19-be9c-4a91-bc2d-b1ff936194da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" Sep 11 14:40:11.230324 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230240 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-var-lib-openvswitch\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.230324 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230232 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa59047c-1d81-44d0-b3a3-9e56bf53511c-tmp-dir\") pod \"node-resolver-ppnhx\" (UID: \"fa59047c-1d81-44d0-b3a3-9e56bf53511c\") " pod="openshift-dns/node-resolver-ppnhx" Sep 11 14:40:11.230324 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230274 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8g5r\" (UniqueName: \"kubernetes.io/projected/ecdbcfa3-6309-46be-a9c9-ea8eb658de34-kube-api-access-p8g5r\") pod \"network-check-target-5hp9w\" (UID: \"ecdbcfa3-6309-46be-a9c9-ea8eb658de34\") " pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:40:11.230324 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230304 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-dbus\") pod \"global-pull-secret-syncer-mlrnk\" (UID: \"a1b523f4-d191-49bb-8e0e-4999e6fa1a92\") " pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:11.230324 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230322 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-var-lib-openvswitch\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.230706 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230328 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-etc-modprobe-d\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.230706 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230350 2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 11 14:40:11.230706 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230352 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-host\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.230706 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230399 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-multus-conf-dir\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.230706 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230431 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-867wm\" (UniqueName: \"kubernetes.io/projected/40743b05-b892-4d5a-bd13-46fb3822ad56-kube-api-access-867wm\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.230706 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230460 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-multus-conf-dir\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.230706 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230548 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f1b03dc4-474b-44b4-ba75-5feb9a72ebd3-iptables-alerter-script\") pod \"iptables-alerter-m7tfj\" (UID: \"f1b03dc4-474b-44b4-ba75-5feb9a72ebd3\") " pod="openshift-network-operator/iptables-alerter-m7tfj" Sep 11 14:40:11.230706 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230607 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-host-run-k8s-cni-cncf-io\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.230706 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230656 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-etc-kubernetes\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.230706 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230682 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-host-var-lib-kubelet\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.231210 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230711 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.231210 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230714 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c0b0394a-1326-4465-b674-23c0c6fbe017-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-75zzz\" (UID: \"c0b0394a-1326-4465-b674-23c0c6fbe017\") " pod="openshift-multus/multus-additional-cni-plugins-75zzz" Sep 11 14:40:11.231210 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230736 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b82ed19-be9c-4a91-bc2d-b1ff936194da-sys-fs\") pod \"aws-ebs-csi-driver-node-h7fcz\" (UID: \"7b82ed19-be9c-4a91-bc2d-b1ff936194da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" Sep 11 14:40:11.231210 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230725 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d39f63c1-32f1-4c2a-8585-a3defd284630-cni-binary-copy\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.231210 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230758 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-etc-kubernetes\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.231210 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230834 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fa59047c-1d81-44d0-b3a3-9e56bf53511c-hosts-file\") pod \"node-resolver-ppnhx\" (UID: \"fa59047c-1d81-44d0-b3a3-9e56bf53511c\") " pod="openshift-dns/node-resolver-ppnhx" Sep 11 14:40:11.231210 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230841 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.231210 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230849 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-host-var-lib-kubelet\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.231210 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230864 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-host-cni-netd\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.231210 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230910 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b82ed19-be9c-4a91-bc2d-b1ff936194da-sys-fs\") pod \"aws-ebs-csi-driver-node-h7fcz\" (UID: \"7b82ed19-be9c-4a91-bc2d-b1ff936194da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" Sep 11 14:40:11.231210 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230918 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0b0394a-1326-4465-b674-23c0c6fbe017-system-cni-dir\") pod \"multus-additional-cni-plugins-75zzz\" (UID: \"c0b0394a-1326-4465-b674-23c0c6fbe017\") " pod="openshift-multus/multus-additional-cni-plugins-75zzz" Sep 11 14:40:11.231210 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230919 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fa59047c-1d81-44d0-b3a3-9e56bf53511c-hosts-file\") pod \"node-resolver-ppnhx\" (UID: \"fa59047c-1d81-44d0-b3a3-9e56bf53511c\") " pod="openshift-dns/node-resolver-ppnhx" Sep 11 14:40:11.231210 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230954 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-host-run-k8s-cni-cncf-io\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.231210 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.230988 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0b0394a-1326-4465-b674-23c0c6fbe017-system-cni-dir\") pod \"multus-additional-cni-plugins-75zzz\" (UID: \"c0b0394a-1326-4465-b674-23c0c6fbe017\") " pod="openshift-multus/multus-additional-cni-plugins-75zzz" Sep 11 14:40:11.231210 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231020 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b82ed19-be9c-4a91-bc2d-b1ff936194da-device-dir\") pod \"aws-ebs-csi-driver-node-h7fcz\" (UID: \"7b82ed19-be9c-4a91-bc2d-b1ff936194da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" Sep 11 14:40:11.231210 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231049 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-systemd-units\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.231210 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231042 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-host-cni-netd\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.231983 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231078 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-etc-systemd\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.231983 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231127 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/804431e5-f91c-4e79-911e-6dcadceeb6ef-tmp\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.231983 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231127 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b82ed19-be9c-4a91-bc2d-b1ff936194da-device-dir\") pod \"aws-ebs-csi-driver-node-h7fcz\" (UID: \"7b82ed19-be9c-4a91-bc2d-b1ff936194da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" Sep 11 14:40:11.231983 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231158 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-systemd-units\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.231983 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231192 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-host-cni-bin\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.231983 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231225 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-host-cni-bin\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.231983 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231228 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-run-systemd\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.231983 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231269 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-host-run-ovn-kubernetes\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.231983 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231271 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f1b03dc4-474b-44b4-ba75-5feb9a72ebd3-iptables-alerter-script\") pod \"iptables-alerter-m7tfj\" (UID: \"f1b03dc4-474b-44b4-ba75-5feb9a72ebd3\") " pod="openshift-network-operator/iptables-alerter-m7tfj" Sep 11 14:40:11.231983 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231318 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-host-run-ovn-kubernetes\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.231983 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231318 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-run-systemd\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.231983 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231344 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ztsw\" (UniqueName: \"kubernetes.io/projected/c0b0394a-1326-4465-b674-23c0c6fbe017-kube-api-access-6ztsw\") pod \"multus-additional-cni-plugins-75zzz\" (UID: \"c0b0394a-1326-4465-b674-23c0c6fbe017\") " pod="openshift-multus/multus-additional-cni-plugins-75zzz" Sep 11 14:40:11.231983 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231363 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1b03dc4-474b-44b4-ba75-5feb9a72ebd3-host-slash\") pod \"iptables-alerter-m7tfj\" (UID: \"f1b03dc4-474b-44b4-ba75-5feb9a72ebd3\") " pod="openshift-network-operator/iptables-alerter-m7tfj" Sep 11 14:40:11.231983 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231387 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9bkd\" (UniqueName: \"kubernetes.io/projected/d39f63c1-32f1-4c2a-8585-a3defd284630-kube-api-access-g9bkd\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.231983 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231405 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ac306f32-44e2-4e36-8e09-2bb71e246eb8-konnectivity-ca\") pod \"konnectivity-agent-vfmp4\" (UID: \"ac306f32-44e2-4e36-8e09-2bb71e246eb8\") " pod="kube-system/konnectivity-agent-vfmp4" Sep 11 14:40:11.231983 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231421 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-kubelet-config\") pod \"global-pull-secret-syncer-mlrnk\" (UID: \"a1b523f4-d191-49bb-8e0e-4999e6fa1a92\") " pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:11.231983 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231438 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-log-socket\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.232831 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231435 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1b03dc4-474b-44b4-ba75-5feb9a72ebd3-host-slash\") pod \"iptables-alerter-m7tfj\" (UID: \"f1b03dc4-474b-44b4-ba75-5feb9a72ebd3\") " pod="openshift-network-operator/iptables-alerter-m7tfj" Sep 11 14:40:11.232831 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231465 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c0b0394a-1326-4465-b674-23c0c6fbe017-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-75zzz\" (UID: \"c0b0394a-1326-4465-b674-23c0c6fbe017\") " pod="openshift-multus/multus-additional-cni-plugins-75zzz" Sep 11 14:40:11.232831 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231511 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-multus-socket-dir-parent\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.232831 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231499 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-log-socket\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.232831 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231543 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73182ce3-cb4b-4fcb-992e-85d116cf7e15-metrics-certs\") pod \"network-metrics-daemon-brx7q\" (UID: \"73182ce3-cb4b-4fcb-992e-85d116cf7e15\") " pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:11.232831 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231571 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-etc-kubernetes\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.232831 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231599 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-sys\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.232831 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231646 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/40743b05-b892-4d5a-bd13-46fb3822ad56-ovnkube-script-lib\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.232831 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231664 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b82ed19-be9c-4a91-bc2d-b1ff936194da-registration-dir\") pod \"aws-ebs-csi-driver-node-h7fcz\" (UID: \"7b82ed19-be9c-4a91-bc2d-b1ff936194da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" Sep 11 14:40:11.232831 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231679 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-etc-sysctl-d\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.232831 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231703 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcrml\" (UniqueName: \"kubernetes.io/projected/804431e5-f91c-4e79-911e-6dcadceeb6ef-kube-api-access-tcrml\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.232831 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231728 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b82ed19-be9c-4a91-bc2d-b1ff936194da-socket-dir\") pod \"aws-ebs-csi-driver-node-h7fcz\" (UID: \"7b82ed19-be9c-4a91-bc2d-b1ff936194da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" Sep 11 14:40:11.232831 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231595 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-multus-socket-dir-parent\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.232831 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231752 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b82ed19-be9c-4a91-bc2d-b1ff936194da-etc-selinux\") pod \"aws-ebs-csi-driver-node-h7fcz\" (UID: \"7b82ed19-be9c-4a91-bc2d-b1ff936194da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" Sep 11 14:40:11.232831 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231775 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b82ed19-be9c-4a91-bc2d-b1ff936194da-registration-dir\") pod \"aws-ebs-csi-driver-node-h7fcz\" (UID: \"7b82ed19-be9c-4a91-bc2d-b1ff936194da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" Sep 11 14:40:11.232831 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231866 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzxlv\" (UniqueName: \"kubernetes.io/projected/fa59047c-1d81-44d0-b3a3-9e56bf53511c-kube-api-access-tzxlv\") pod \"node-resolver-ppnhx\" (UID: \"fa59047c-1d81-44d0-b3a3-9e56bf53511c\") " pod="openshift-dns/node-resolver-ppnhx" Sep 11 14:40:11.232831 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231870 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b82ed19-be9c-4a91-bc2d-b1ff936194da-etc-selinux\") pod \"aws-ebs-csi-driver-node-h7fcz\" (UID: \"7b82ed19-be9c-4a91-bc2d-b1ff936194da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" Sep 11 14:40:11.233395 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231905 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-run-ovn\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.233395 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231939 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mvxl\" (UniqueName: \"kubernetes.io/projected/f1b03dc4-474b-44b4-ba75-5feb9a72ebd3-kube-api-access-2mvxl\") pod \"iptables-alerter-m7tfj\" (UID: \"f1b03dc4-474b-44b4-ba75-5feb9a72ebd3\") " pod="openshift-network-operator/iptables-alerter-m7tfj" Sep 11 14:40:11.233395 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231996 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-run-ovn\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.233395 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.231969 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d39f63c1-32f1-4c2a-8585-a3defd284630-multus-daemon-config\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.233395 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232029 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b82ed19-be9c-4a91-bc2d-b1ff936194da-socket-dir\") pod \"aws-ebs-csi-driver-node-h7fcz\" (UID: \"7b82ed19-be9c-4a91-bc2d-b1ff936194da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" Sep 11 14:40:11.233395 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232036 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c0b0394a-1326-4465-b674-23c0c6fbe017-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-75zzz\" (UID: \"c0b0394a-1326-4465-b674-23c0c6fbe017\") " pod="openshift-multus/multus-additional-cni-plugins-75zzz" Sep 11 14:40:11.233395 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232058 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-host-kubelet\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.233395 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232136 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-host-kubelet\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.233395 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232299 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0b0394a-1326-4465-b674-23c0c6fbe017-cnibin\") pod \"multus-additional-cni-plugins-75zzz\" (UID: \"c0b0394a-1326-4465-b674-23c0c6fbe017\") " pod="openshift-multus/multus-additional-cni-plugins-75zzz" Sep 11 14:40:11.233395 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232374 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-multus-cni-dir\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.233395 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232399 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0b0394a-1326-4465-b674-23c0c6fbe017-cnibin\") pod \"multus-additional-cni-plugins-75zzz\" (UID: \"c0b0394a-1326-4465-b674-23c0c6fbe017\") " pod="openshift-multus/multus-additional-cni-plugins-75zzz" Sep 11 14:40:11.233395 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232406 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b82ed19-be9c-4a91-bc2d-b1ff936194da-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h7fcz\" (UID: \"7b82ed19-be9c-4a91-bc2d-b1ff936194da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" Sep 11 14:40:11.233395 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232437 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/40743b05-b892-4d5a-bd13-46fb3822ad56-ovnkube-script-lib\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.233395 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232457 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b82ed19-be9c-4a91-bc2d-b1ff936194da-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h7fcz\" (UID: \"7b82ed19-be9c-4a91-bc2d-b1ff936194da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" Sep 11 14:40:11.233395 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232464 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-run-openvswitch\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.233395 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232498 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/40743b05-b892-4d5a-bd13-46fb3822ad56-ovnkube-config\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.233395 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232513 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-run-openvswitch\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.233900 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232526 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/40743b05-b892-4d5a-bd13-46fb3822ad56-env-overrides\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.233900 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232551 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-multus-cni-dir\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.233900 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232527 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d39f63c1-32f1-4c2a-8585-a3defd284630-multus-daemon-config\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.233900 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232581 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0b0394a-1326-4465-b674-23c0c6fbe017-cni-binary-copy\") pod \"multus-additional-cni-plugins-75zzz\" (UID: \"c0b0394a-1326-4465-b674-23c0c6fbe017\") " pod="openshift-multus/multus-additional-cni-plugins-75zzz" Sep 11 14:40:11.233900 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232614 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0b0394a-1326-4465-b674-23c0c6fbe017-tuning-conf-dir\") pod \"multus-additional-cni-plugins-75zzz\" (UID: \"c0b0394a-1326-4465-b674-23c0c6fbe017\") " pod="openshift-multus/multus-additional-cni-plugins-75zzz" Sep 11 14:40:11.233900 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232629 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-host-run-netns\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.233900 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232676 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-etc-sysctl-conf\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.233900 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232690 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-etc-openvswitch\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.233900 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232704 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-hostroot\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.233900 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232723 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-host-run-multus-certs\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.233900 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232786 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-host-run-netns\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.233900 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232850 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40743b05-b892-4d5a-bd13-46fb3822ad56-etc-openvswitch\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.233900 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232870 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0b0394a-1326-4465-b674-23c0c6fbe017-tuning-conf-dir\") pod \"multus-additional-cni-plugins-75zzz\" (UID: \"c0b0394a-1326-4465-b674-23c0c6fbe017\") " pod="openshift-multus/multus-additional-cni-plugins-75zzz" Sep 11 14:40:11.233900 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232894 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-hostroot\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.233900 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232895 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ac306f32-44e2-4e36-8e09-2bb71e246eb8-agent-certs\") pod \"konnectivity-agent-vfmp4\" (UID: \"ac306f32-44e2-4e36-8e09-2bb71e246eb8\") " pod="kube-system/konnectivity-agent-vfmp4" Sep 11 14:40:11.233900 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232929 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d39f63c1-32f1-4c2a-8585-a3defd284630-host-run-multus-certs\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.233900 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.232998 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/40743b05-b892-4d5a-bd13-46fb3822ad56-env-overrides\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.233900 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.233113 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0b0394a-1326-4465-b674-23c0c6fbe017-cni-binary-copy\") pod \"multus-additional-cni-plugins-75zzz\" (UID: \"c0b0394a-1326-4465-b674-23c0c6fbe017\") " pod="openshift-multus/multus-additional-cni-plugins-75zzz" Sep 11 14:40:11.234687 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.233130 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/40743b05-b892-4d5a-bd13-46fb3822ad56-ovnkube-config\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.234687 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.234407 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/40743b05-b892-4d5a-bd13-46fb3822ad56-ovn-node-metrics-cert\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.238905 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:11.238882 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:40:11.238989 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:11.238911 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:40:11.238989 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:11.238926 2583 projected.go:196] Error preparing data for projected volume kube-api-access-p8g5r for pod openshift-network-diagnostics/network-check-target-5hp9w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:40:11.239111 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:11.239091 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecdbcfa3-6309-46be-a9c9-ea8eb658de34-kube-api-access-p8g5r podName:ecdbcfa3-6309-46be-a9c9-ea8eb658de34 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:11.73906995 +0000 UTC m=+2.983055603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-p8g5r" (UniqueName: "kubernetes.io/projected/ecdbcfa3-6309-46be-a9c9-ea8eb658de34-kube-api-access-p8g5r") pod "network-check-target-5hp9w" (UID: "ecdbcfa3-6309-46be-a9c9-ea8eb658de34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:40:11.241818 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.241744 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mvxl\" (UniqueName: \"kubernetes.io/projected/f1b03dc4-474b-44b4-ba75-5feb9a72ebd3-kube-api-access-2mvxl\") pod \"iptables-alerter-m7tfj\" (UID: \"f1b03dc4-474b-44b4-ba75-5feb9a72ebd3\") " pod="openshift-network-operator/iptables-alerter-m7tfj" Sep 11 14:40:11.242402 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.242347 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx4t5\" (UniqueName: \"kubernetes.io/projected/7b82ed19-be9c-4a91-bc2d-b1ff936194da-kube-api-access-lx4t5\") pod \"aws-ebs-csi-driver-node-h7fcz\" (UID: \"7b82ed19-be9c-4a91-bc2d-b1ff936194da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" Sep 11 14:40:11.242503 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.242489 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9bkd\" (UniqueName: \"kubernetes.io/projected/d39f63c1-32f1-4c2a-8585-a3defd284630-kube-api-access-g9bkd\") pod \"multus-97b8j\" (UID: \"d39f63c1-32f1-4c2a-8585-a3defd284630\") " pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.242564 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.242536 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzxlv\" (UniqueName: \"kubernetes.io/projected/fa59047c-1d81-44d0-b3a3-9e56bf53511c-kube-api-access-tzxlv\") pod \"node-resolver-ppnhx\" (UID: \"fa59047c-1d81-44d0-b3a3-9e56bf53511c\") " pod="openshift-dns/node-resolver-ppnhx" Sep 11 14:40:11.243303 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.243147 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ztsw\" (UniqueName: \"kubernetes.io/projected/c0b0394a-1326-4465-b674-23c0c6fbe017-kube-api-access-6ztsw\") pod \"multus-additional-cni-plugins-75zzz\" (UID: \"c0b0394a-1326-4465-b674-23c0c6fbe017\") " pod="openshift-multus/multus-additional-cni-plugins-75zzz" Sep 11 14:40:11.243840 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.243818 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-867wm\" (UniqueName: \"kubernetes.io/projected/40743b05-b892-4d5a-bd13-46fb3822ad56-kube-api-access-867wm\") pod \"ovnkube-node-vlg7j\" (UID: \"40743b05-b892-4d5a-bd13-46fb3822ad56\") " pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.324864 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.324837 2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:40:11.333330 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.333301 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7prq\" (UniqueName: \"kubernetes.io/projected/73182ce3-cb4b-4fcb-992e-85d116cf7e15-kube-api-access-q7prq\") pod \"network-metrics-daemon-brx7q\" (UID: \"73182ce3-cb4b-4fcb-992e-85d116cf7e15\") " pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:11.333460 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.333337 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8dc4e60b-0baa-45d8-93b2-63436c43e3be-host\") pod \"node-ca-4vlml\" (UID: \"8dc4e60b-0baa-45d8-93b2-63436c43e3be\") " pod="openshift-image-registry/node-ca-4vlml" Sep 11 14:40:11.333460 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.333363 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8dc4e60b-0baa-45d8-93b2-63436c43e3be-serviceca\") pod \"node-ca-4vlml\" (UID: \"8dc4e60b-0baa-45d8-93b2-63436c43e3be\") " pod="openshift-image-registry/node-ca-4vlml" Sep 11 14:40:11.333460 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.333391 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-675dv\" (UniqueName: \"kubernetes.io/projected/8dc4e60b-0baa-45d8-93b2-63436c43e3be-kube-api-access-675dv\") pod \"node-ca-4vlml\" (UID: \"8dc4e60b-0baa-45d8-93b2-63436c43e3be\") " pod="openshift-image-registry/node-ca-4vlml" Sep 11 14:40:11.333460 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.333408 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-etc-sysconfig\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.333460 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.333449 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8dc4e60b-0baa-45d8-93b2-63436c43e3be-host\") pod \"node-ca-4vlml\" (UID: \"8dc4e60b-0baa-45d8-93b2-63436c43e3be\") " pod="openshift-image-registry/node-ca-4vlml" Sep 11 14:40:11.333722 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.333462 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-etc-sysconfig\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.333722 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.333612 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-lib-modules\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.333722 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.333650 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/804431e5-f91c-4e79-911e-6dcadceeb6ef-etc-tuned\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.333722 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.333678 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-var-lib-kubelet\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.333722 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.333710 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-dbus\") pod \"global-pull-secret-syncer-mlrnk\" (UID: \"a1b523f4-d191-49bb-8e0e-4999e6fa1a92\") " pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:11.334000 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.333731 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-etc-modprobe-d\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.334000 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.333750 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-host\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.334000 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.333795 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-etc-systemd\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.334000 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.333799 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-lib-modules\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.334000 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.333816 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/804431e5-f91c-4e79-911e-6dcadceeb6ef-tmp\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.334000 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.333845 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ac306f32-44e2-4e36-8e09-2bb71e246eb8-konnectivity-ca\") pod \"konnectivity-agent-vfmp4\" (UID: \"ac306f32-44e2-4e36-8e09-2bb71e246eb8\") " pod="kube-system/konnectivity-agent-vfmp4" Sep 11 14:40:11.334000 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.333869 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-kubelet-config\") pod \"global-pull-secret-syncer-mlrnk\" (UID: \"a1b523f4-d191-49bb-8e0e-4999e6fa1a92\") " pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:11.334000 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.333897 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73182ce3-cb4b-4fcb-992e-85d116cf7e15-metrics-certs\") pod \"network-metrics-daemon-brx7q\" (UID: \"73182ce3-cb4b-4fcb-992e-85d116cf7e15\") " pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:11.334000 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.333904 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-etc-modprobe-d\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.334000 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.333922 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-etc-kubernetes\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.334000 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.333945 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-sys\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.334000 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.333959 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-var-lib-kubelet\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.334000 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.333980 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8dc4e60b-0baa-45d8-93b2-63436c43e3be-serviceca\") pod \"node-ca-4vlml\" (UID: \"8dc4e60b-0baa-45d8-93b2-63436c43e3be\") " pod="openshift-image-registry/node-ca-4vlml" Sep 11 14:40:11.334649 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.334008 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-dbus\") pod \"global-pull-secret-syncer-mlrnk\" (UID: \"a1b523f4-d191-49bb-8e0e-4999e6fa1a92\") " pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:11.334649 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.334051 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-host\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.334649 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.334055 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-sys\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.334649 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.334116 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-etc-kubernetes\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.334649 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.334119 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-etc-systemd\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.334649 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.334154 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-etc-sysctl-d\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.334649 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.334164 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-kubelet-config\") pod \"global-pull-secret-syncer-mlrnk\" (UID: \"a1b523f4-d191-49bb-8e0e-4999e6fa1a92\") " pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:11.334649 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:11.334168 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:40:11.334649 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.334193 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcrml\" (UniqueName: \"kubernetes.io/projected/804431e5-f91c-4e79-911e-6dcadceeb6ef-kube-api-access-tcrml\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.334649 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.334234 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-etc-sysctl-conf\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.334649 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:11.334248 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73182ce3-cb4b-4fcb-992e-85d116cf7e15-metrics-certs podName:73182ce3-cb4b-4fcb-992e-85d116cf7e15 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:11.834229204 +0000 UTC m=+3.078214863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73182ce3-cb4b-4fcb-992e-85d116cf7e15-metrics-certs") pod "network-metrics-daemon-brx7q" (UID: "73182ce3-cb4b-4fcb-992e-85d116cf7e15") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:40:11.334649 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.334271 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-etc-sysctl-d\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.334649 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.334288 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ac306f32-44e2-4e36-8e09-2bb71e246eb8-agent-certs\") pod \"konnectivity-agent-vfmp4\" (UID: \"ac306f32-44e2-4e36-8e09-2bb71e246eb8\") " pod="kube-system/konnectivity-agent-vfmp4" Sep 11 14:40:11.334649 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.334323 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-original-pull-secret\") pod \"global-pull-secret-syncer-mlrnk\" (UID: \"a1b523f4-d191-49bb-8e0e-4999e6fa1a92\") " pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:11.334649 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.334353 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-run\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.334649 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.334357 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-etc-sysctl-conf\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.334649 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:11.334470 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:40:11.334649 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.334472 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ac306f32-44e2-4e36-8e09-2bb71e246eb8-konnectivity-ca\") pod \"konnectivity-agent-vfmp4\" (UID: \"ac306f32-44e2-4e36-8e09-2bb71e246eb8\") " pod="kube-system/konnectivity-agent-vfmp4" Sep 11 14:40:11.335580 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.334490 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/804431e5-f91c-4e79-911e-6dcadceeb6ef-run\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.335580 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:11.334514 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-original-pull-secret podName:a1b523f4-d191-49bb-8e0e-4999e6fa1a92 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:11.834499085 +0000 UTC m=+3.078484727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-original-pull-secret") pod "global-pull-secret-syncer-mlrnk" (UID: "a1b523f4-d191-49bb-8e0e-4999e6fa1a92") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:40:11.336837 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.336816 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/804431e5-f91c-4e79-911e-6dcadceeb6ef-etc-tuned\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.338145 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.338123 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ac306f32-44e2-4e36-8e09-2bb71e246eb8-agent-certs\") pod \"konnectivity-agent-vfmp4\" (UID: \"ac306f32-44e2-4e36-8e09-2bb71e246eb8\") " pod="kube-system/konnectivity-agent-vfmp4" Sep 11 14:40:11.338313 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.338293 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/804431e5-f91c-4e79-911e-6dcadceeb6ef-tmp\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.347333 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.347265 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7prq\" (UniqueName: \"kubernetes.io/projected/73182ce3-cb4b-4fcb-992e-85d116cf7e15-kube-api-access-q7prq\") pod \"network-metrics-daemon-brx7q\" (UID: \"73182ce3-cb4b-4fcb-992e-85d116cf7e15\") " pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:11.349158 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.349118 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcrml\" (UniqueName: \"kubernetes.io/projected/804431e5-f91c-4e79-911e-6dcadceeb6ef-kube-api-access-tcrml\") pod \"tuned-7mbt2\" (UID: \"804431e5-f91c-4e79-911e-6dcadceeb6ef\") " pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.349246 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.349165 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-675dv\" (UniqueName: \"kubernetes.io/projected/8dc4e60b-0baa-45d8-93b2-63436c43e3be-kube-api-access-675dv\") pod \"node-ca-4vlml\" (UID: \"8dc4e60b-0baa-45d8-93b2-63436c43e3be\") " pod="openshift-image-registry/node-ca-4vlml" Sep 11 14:40:11.430303 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.430261 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" Sep 11 14:40:11.440445 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:11.440412 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b82ed19_be9c_4a91_bc2d_b1ff936194da.slice/crio-50312e83a042f367b925c92ab42cbe959a6fd7f925b6c92bee5416361697e547 WatchSource:0}: Error finding container 50312e83a042f367b925c92ab42cbe959a6fd7f925b6c92bee5416361697e547: Status 404 returned error can't find the container with id 50312e83a042f367b925c92ab42cbe959a6fd7f925b6c92bee5416361697e547 Sep 11 14:40:11.442576 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.442331 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m7tfj" Sep 11 14:40:11.452765 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:11.452735 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b03dc4_474b_44b4_ba75_5feb9a72ebd3.slice/crio-3c8f65c5eb3402c83f5aff36b6057e5cf6b17599544b98ebf86ebfc73a95a9f7 WatchSource:0}: Error finding container 3c8f65c5eb3402c83f5aff36b6057e5cf6b17599544b98ebf86ebfc73a95a9f7: Status 404 returned error can't find the container with id 3c8f65c5eb3402c83f5aff36b6057e5cf6b17599544b98ebf86ebfc73a95a9f7 Sep 11 14:40:11.453633 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.453612 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ppnhx" Sep 11 14:40:11.459821 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.459762 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:11.462853 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:11.462801 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa59047c_1d81_44d0_b3a3_9e56bf53511c.slice/crio-3b0b5fd8e8d57b483eb55d10c5a8dfc4a01b3c6efc4ff3c09f846d3d09dfb6c6 WatchSource:0}: Error finding container 3b0b5fd8e8d57b483eb55d10c5a8dfc4a01b3c6efc4ff3c09f846d3d09dfb6c6: Status 404 returned error can't find the container with id 3b0b5fd8e8d57b483eb55d10c5a8dfc4a01b3c6efc4ff3c09f846d3d09dfb6c6 Sep 11 14:40:11.468758 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.468709 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-97b8j" Sep 11 14:40:11.478128 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.478102 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-75zzz" Sep 11 14:40:11.487206 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.487184 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vfmp4" Sep 11 14:40:11.495989 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.495950 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" Sep 11 14:40:11.503624 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.503603 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4vlml" Sep 11 14:40:11.839065 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.838998 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-original-pull-secret\") pod \"global-pull-secret-syncer-mlrnk\" (UID: \"a1b523f4-d191-49bb-8e0e-4999e6fa1a92\") " pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:11.839065 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.839048 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8g5r\" (UniqueName: \"kubernetes.io/projected/ecdbcfa3-6309-46be-a9c9-ea8eb658de34-kube-api-access-p8g5r\") pod \"network-check-target-5hp9w\" (UID: \"ecdbcfa3-6309-46be-a9c9-ea8eb658de34\") " pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:40:11.839255 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:11.839095 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73182ce3-cb4b-4fcb-992e-85d116cf7e15-metrics-certs\") pod \"network-metrics-daemon-brx7q\" (UID: \"73182ce3-cb4b-4fcb-992e-85d116cf7e15\") " pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:11.839255 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:11.839164 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:40:11.839255 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:11.839199 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:40:11.839701 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:11.839585 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:40:11.839701 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:11.839616 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:40:11.839701 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:11.839622 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-original-pull-secret podName:a1b523f4-d191-49bb-8e0e-4999e6fa1a92 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:12.839213014 +0000 UTC m=+4.083198652 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-original-pull-secret") pod "global-pull-secret-syncer-mlrnk" (UID: "a1b523f4-d191-49bb-8e0e-4999e6fa1a92") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:40:11.839701 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:11.839642 2583 projected.go:196] Error preparing data for projected volume kube-api-access-p8g5r for pod openshift-network-diagnostics/network-check-target-5hp9w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:40:11.839701 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:11.839663 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73182ce3-cb4b-4fcb-992e-85d116cf7e15-metrics-certs podName:73182ce3-cb4b-4fcb-992e-85d116cf7e15 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:12.839641738 +0000 UTC m=+4.083627389 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73182ce3-cb4b-4fcb-992e-85d116cf7e15-metrics-certs") pod "network-metrics-daemon-brx7q" (UID: "73182ce3-cb4b-4fcb-992e-85d116cf7e15") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:40:11.839701 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:11.839706 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecdbcfa3-6309-46be-a9c9-ea8eb658de34-kube-api-access-p8g5r podName:ecdbcfa3-6309-46be-a9c9-ea8eb658de34 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:12.839679731 +0000 UTC m=+4.083665375 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-p8g5r" (UniqueName: "kubernetes.io/projected/ecdbcfa3-6309-46be-a9c9-ea8eb658de34-kube-api-access-p8g5r") pod "network-check-target-5hp9w" (UID: "ecdbcfa3-6309-46be-a9c9-ea8eb658de34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:40:12.173824 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:12.173746 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 14:35:11 +0000 UTC" deadline="2028-05-31 17:43:09.966691804 +0000 UTC" Sep 11 14:40:12.173824 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:12.173818 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15075h2m57.792880043s" Sep 11 14:40:12.333227 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:12.333185 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ppnhx" event={"ID":"fa59047c-1d81-44d0-b3a3-9e56bf53511c","Type":"ContainerStarted","Data":"3b0b5fd8e8d57b483eb55d10c5a8dfc4a01b3c6efc4ff3c09f846d3d09dfb6c6"} Sep 11 14:40:12.335581 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:12.335520 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m7tfj" event={"ID":"f1b03dc4-474b-44b4-ba75-5feb9a72ebd3","Type":"ContainerStarted","Data":"3c8f65c5eb3402c83f5aff36b6057e5cf6b17599544b98ebf86ebfc73a95a9f7"} Sep 11 14:40:12.337146 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:12.337089 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" event={"ID":"7b82ed19-be9c-4a91-bc2d-b1ff936194da","Type":"ContainerStarted","Data":"50312e83a042f367b925c92ab42cbe959a6fd7f925b6c92bee5416361697e547"} Sep 11 14:40:12.559267 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:12.559183 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40743b05_b892_4d5a_bd13_46fb3822ad56.slice/crio-2cc40306a88435da4ffee10ba98f010b7208a6d3aa47fdf08bbece0cdb9a3d5f WatchSource:0}: Error finding container 2cc40306a88435da4ffee10ba98f010b7208a6d3aa47fdf08bbece0cdb9a3d5f: Status 404 returned error can't find the container with id 2cc40306a88435da4ffee10ba98f010b7208a6d3aa47fdf08bbece0cdb9a3d5f Sep 11 14:40:12.585227 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:12.585197 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b0394a_1326_4465_b674_23c0c6fbe017.slice/crio-64b0c59670e1c29470324cd0bca21c27c530a243ef682f4a5ed6a965b3030ccd WatchSource:0}: Error finding container 64b0c59670e1c29470324cd0bca21c27c530a243ef682f4a5ed6a965b3030ccd: Status 404 returned error can't find the container with id 64b0c59670e1c29470324cd0bca21c27c530a243ef682f4a5ed6a965b3030ccd Sep 11 14:40:12.587001 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:12.586974 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod804431e5_f91c_4e79_911e_6dcadceeb6ef.slice/crio-ffe8a08444da79991a5b2b3b406734fa86de59ade1219737bc8c10c909cfcdfb WatchSource:0}: Error finding container ffe8a08444da79991a5b2b3b406734fa86de59ade1219737bc8c10c909cfcdfb: Status 404 returned error can't find the container with id ffe8a08444da79991a5b2b3b406734fa86de59ade1219737bc8c10c909cfcdfb Sep 11 14:40:12.611470 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:12.611445 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dc4e60b_0baa_45d8_93b2_63436c43e3be.slice/crio-ef1d32963508b253d9b129782f142382c7575c81c2d8d6298fc80f62cc2ae3de WatchSource:0}: Error finding container ef1d32963508b253d9b129782f142382c7575c81c2d8d6298fc80f62cc2ae3de: Status 404 returned error can't find the container with id ef1d32963508b253d9b129782f142382c7575c81c2d8d6298fc80f62cc2ae3de Sep 11 14:40:12.612877 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:12.612856 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac306f32_44e2_4e36_8e09_2bb71e246eb8.slice/crio-36d729a215f9787435de5315f7282615e05a0c5316013dfea50f85890b90c115 WatchSource:0}: Error finding container 36d729a215f9787435de5315f7282615e05a0c5316013dfea50f85890b90c115: Status 404 returned error can't find the container with id 36d729a215f9787435de5315f7282615e05a0c5316013dfea50f85890b90c115 Sep 11 14:40:12.846295 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:12.845353 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8g5r\" (UniqueName: \"kubernetes.io/projected/ecdbcfa3-6309-46be-a9c9-ea8eb658de34-kube-api-access-p8g5r\") pod \"network-check-target-5hp9w\" (UID: \"ecdbcfa3-6309-46be-a9c9-ea8eb658de34\") " pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:40:12.846295 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:12.845415 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73182ce3-cb4b-4fcb-992e-85d116cf7e15-metrics-certs\") pod \"network-metrics-daemon-brx7q\" (UID: \"73182ce3-cb4b-4fcb-992e-85d116cf7e15\") " pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:12.846295 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:12.845461 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-original-pull-secret\") pod \"global-pull-secret-syncer-mlrnk\" (UID: \"a1b523f4-d191-49bb-8e0e-4999e6fa1a92\") " pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:12.846295 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:12.845596 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:40:12.846295 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:12.845656 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-original-pull-secret podName:a1b523f4-d191-49bb-8e0e-4999e6fa1a92 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:14.845642713 +0000 UTC m=+6.089628351 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-original-pull-secret") pod "global-pull-secret-syncer-mlrnk" (UID: "a1b523f4-d191-49bb-8e0e-4999e6fa1a92") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:40:12.846295 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:12.845899 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:40:12.846295 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:12.845952 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73182ce3-cb4b-4fcb-992e-85d116cf7e15-metrics-certs podName:73182ce3-cb4b-4fcb-992e-85d116cf7e15 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:14.845937997 +0000 UTC m=+6.089923649 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73182ce3-cb4b-4fcb-992e-85d116cf7e15-metrics-certs") pod "network-metrics-daemon-brx7q" (UID: "73182ce3-cb4b-4fcb-992e-85d116cf7e15") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:40:12.846295 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:12.846025 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:40:12.846295 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:12.846037 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:40:12.846295 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:12.846049 2583 projected.go:196] Error preparing data for projected volume kube-api-access-p8g5r for pod openshift-network-diagnostics/network-check-target-5hp9w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:40:12.846295 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:12.846078 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecdbcfa3-6309-46be-a9c9-ea8eb658de34-kube-api-access-p8g5r podName:ecdbcfa3-6309-46be-a9c9-ea8eb658de34 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:14.846068891 +0000 UTC m=+6.090054544 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-p8g5r" (UniqueName: "kubernetes.io/projected/ecdbcfa3-6309-46be-a9c9-ea8eb658de34-kube-api-access-p8g5r") pod "network-check-target-5hp9w" (UID: "ecdbcfa3-6309-46be-a9c9-ea8eb658de34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:40:13.174955 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:13.174847 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 14:35:11 +0000 UTC" deadline="2028-02-18 02:50:07.131661325 +0000 UTC" Sep 11 14:40:13.174955 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:13.174887 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12588h9m53.956781255s" Sep 11 14:40:13.327374 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:13.327344 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:13.327527 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:13.327485 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlrnk" podUID="a1b523f4-d191-49bb-8e0e-4999e6fa1a92" Sep 11 14:40:13.327915 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:13.327895 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:40:13.328014 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:13.328000 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hp9w" podUID="ecdbcfa3-6309-46be-a9c9-ea8eb658de34" Sep 11 14:40:13.328084 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:13.328073 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:13.328185 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:13.328165 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brx7q" podUID="73182ce3-cb4b-4fcb-992e-85d116cf7e15" Sep 11 14:40:13.345311 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:13.345278 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4vlml" event={"ID":"8dc4e60b-0baa-45d8-93b2-63436c43e3be","Type":"ContainerStarted","Data":"ef1d32963508b253d9b129782f142382c7575c81c2d8d6298fc80f62cc2ae3de"} Sep 11 14:40:13.351199 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:13.351124 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-39.ec2.internal" event={"ID":"34be09161f9ee3edbfcdbbe22b0b8213","Type":"ContainerStarted","Data":"c68262152e0467dfc67b5a4597db6f0dbaa129ad18765d38dbaa7a51e94b0d60"} Sep 11 14:40:13.355860 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:13.355833 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vfmp4" event={"ID":"ac306f32-44e2-4e36-8e09-2bb71e246eb8","Type":"ContainerStarted","Data":"36d729a215f9787435de5315f7282615e05a0c5316013dfea50f85890b90c115"} Sep 11 14:40:13.361546 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:13.361521 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" event={"ID":"804431e5-f91c-4e79-911e-6dcadceeb6ef","Type":"ContainerStarted","Data":"ffe8a08444da79991a5b2b3b406734fa86de59ade1219737bc8c10c909cfcdfb"} Sep 11 14:40:13.369575 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:13.369525 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-39.ec2.internal" podStartSLOduration=2.369506099 podStartE2EDuration="2.369506099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:13.368226588 +0000 UTC m=+4.612212245" watchObservedRunningTime="2026-09-11 14:40:13.369506099 +0000 UTC m=+4.613491762" Sep 11 14:40:13.386279 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:13.386250 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75zzz" event={"ID":"c0b0394a-1326-4465-b674-23c0c6fbe017","Type":"ContainerStarted","Data":"64b0c59670e1c29470324cd0bca21c27c530a243ef682f4a5ed6a965b3030ccd"} Sep 11 14:40:13.398541 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:13.398509 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-97b8j" event={"ID":"d39f63c1-32f1-4c2a-8585-a3defd284630","Type":"ContainerStarted","Data":"c21540e30545f480a7f30e1ca7b9869446106536dd177e70ef22321d972486da"} Sep 11 14:40:13.405409 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:13.405381 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" event={"ID":"40743b05-b892-4d5a-bd13-46fb3822ad56","Type":"ContainerStarted","Data":"2cc40306a88435da4ffee10ba98f010b7208a6d3aa47fdf08bbece0cdb9a3d5f"} Sep 11 14:40:13.430336 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:13.430264 2583 generic.go:356] "Generic (PLEG): container finished" podID="a6e4f2dac72fee1f4d30b95dd4ee38cb" containerID="5ce60f2fb135364ce678c3da04bb3b564b65e32d6bf9e754342b34e3400bc278" exitCode=0 Sep 11 14:40:13.430336 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:13.430311 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-39.ec2.internal" event={"ID":"a6e4f2dac72fee1f4d30b95dd4ee38cb","Type":"ContainerDied","Data":"5ce60f2fb135364ce678c3da04bb3b564b65e32d6bf9e754342b34e3400bc278"} Sep 11 14:40:14.442899 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:14.442850 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-39.ec2.internal" event={"ID":"a6e4f2dac72fee1f4d30b95dd4ee38cb","Type":"ContainerStarted","Data":"6fab61a902cc4eb5acbcf140fd59919570def8f46fc685d5c129f9ac6abd861d"} Sep 11 14:40:14.861373 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:14.861328 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73182ce3-cb4b-4fcb-992e-85d116cf7e15-metrics-certs\") pod \"network-metrics-daemon-brx7q\" (UID: \"73182ce3-cb4b-4fcb-992e-85d116cf7e15\") " pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:14.861373 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:14.861401 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-original-pull-secret\") pod \"global-pull-secret-syncer-mlrnk\" (UID: \"a1b523f4-d191-49bb-8e0e-4999e6fa1a92\") " pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:14.861703 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:14.861445 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8g5r\" (UniqueName: \"kubernetes.io/projected/ecdbcfa3-6309-46be-a9c9-ea8eb658de34-kube-api-access-p8g5r\") pod \"network-check-target-5hp9w\" (UID: \"ecdbcfa3-6309-46be-a9c9-ea8eb658de34\") " pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:40:14.861703 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:14.861552 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:40:14.861703 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:14.861567 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:40:14.861703 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:14.861592 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:40:14.861703 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:14.861605 2583 projected.go:196] Error preparing data for projected volume kube-api-access-p8g5r for pod openshift-network-diagnostics/network-check-target-5hp9w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:40:14.861703 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:14.861628 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73182ce3-cb4b-4fcb-992e-85d116cf7e15-metrics-certs podName:73182ce3-cb4b-4fcb-992e-85d116cf7e15 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:18.861608692 +0000 UTC m=+10.105594336 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73182ce3-cb4b-4fcb-992e-85d116cf7e15-metrics-certs") pod "network-metrics-daemon-brx7q" (UID: "73182ce3-cb4b-4fcb-992e-85d116cf7e15") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:40:14.861703 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:14.861653 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecdbcfa3-6309-46be-a9c9-ea8eb658de34-kube-api-access-p8g5r podName:ecdbcfa3-6309-46be-a9c9-ea8eb658de34 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:18.86163903 +0000 UTC m=+10.105624683 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-p8g5r" (UniqueName: "kubernetes.io/projected/ecdbcfa3-6309-46be-a9c9-ea8eb658de34-kube-api-access-p8g5r") pod "network-check-target-5hp9w" (UID: "ecdbcfa3-6309-46be-a9c9-ea8eb658de34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:40:14.861703 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:14.861552 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:40:14.862169 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:14.861731 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-original-pull-secret podName:a1b523f4-d191-49bb-8e0e-4999e6fa1a92 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:18.861699576 +0000 UTC m=+10.105685218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-original-pull-secret") pod "global-pull-secret-syncer-mlrnk" (UID: "a1b523f4-d191-49bb-8e0e-4999e6fa1a92") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:40:15.324956 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:15.324877 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:15.325127 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:15.325040 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brx7q" podUID="73182ce3-cb4b-4fcb-992e-85d116cf7e15" Sep 11 14:40:15.325485 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:15.325463 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:15.325587 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:15.325563 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlrnk" podUID="a1b523f4-d191-49bb-8e0e-4999e6fa1a92" Sep 11 14:40:15.325776 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:15.325758 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:40:15.325887 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:15.325858 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hp9w" podUID="ecdbcfa3-6309-46be-a9c9-ea8eb658de34" Sep 11 14:40:17.325463 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:17.324986 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:40:17.325463 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:17.325111 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hp9w" podUID="ecdbcfa3-6309-46be-a9c9-ea8eb658de34" Sep 11 14:40:17.325463 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:17.325465 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:17.326030 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:17.325568 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brx7q" podUID="73182ce3-cb4b-4fcb-992e-85d116cf7e15" Sep 11 14:40:17.326030 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:17.325697 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:17.326030 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:17.325787 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlrnk" podUID="a1b523f4-d191-49bb-8e0e-4999e6fa1a92" Sep 11 14:40:18.892329 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:18.892233 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8g5r\" (UniqueName: \"kubernetes.io/projected/ecdbcfa3-6309-46be-a9c9-ea8eb658de34-kube-api-access-p8g5r\") pod \"network-check-target-5hp9w\" (UID: \"ecdbcfa3-6309-46be-a9c9-ea8eb658de34\") " pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:40:18.892329 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:18.892311 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73182ce3-cb4b-4fcb-992e-85d116cf7e15-metrics-certs\") pod \"network-metrics-daemon-brx7q\" (UID: \"73182ce3-cb4b-4fcb-992e-85d116cf7e15\") " pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:18.892873 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:18.892354 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-original-pull-secret\") pod \"global-pull-secret-syncer-mlrnk\" (UID: \"a1b523f4-d191-49bb-8e0e-4999e6fa1a92\") " pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:18.892873 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:18.892403 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:40:18.892873 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:18.892434 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:40:18.892873 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:18.892448 2583 projected.go:196] Error preparing data for projected volume kube-api-access-p8g5r for pod openshift-network-diagnostics/network-check-target-5hp9w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:40:18.892873 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:18.892454 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:40:18.892873 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:18.892506 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecdbcfa3-6309-46be-a9c9-ea8eb658de34-kube-api-access-p8g5r podName:ecdbcfa3-6309-46be-a9c9-ea8eb658de34 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:26.892489957 +0000 UTC m=+18.136475596 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-p8g5r" (UniqueName: "kubernetes.io/projected/ecdbcfa3-6309-46be-a9c9-ea8eb658de34-kube-api-access-p8g5r") pod "network-check-target-5hp9w" (UID: "ecdbcfa3-6309-46be-a9c9-ea8eb658de34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:40:18.892873 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:18.892525 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-original-pull-secret podName:a1b523f4-d191-49bb-8e0e-4999e6fa1a92 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:26.892515842 +0000 UTC m=+18.136501486 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-original-pull-secret") pod "global-pull-secret-syncer-mlrnk" (UID: "a1b523f4-d191-49bb-8e0e-4999e6fa1a92") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:40:18.892873 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:18.892540 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:40:18.892873 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:18.892579 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73182ce3-cb4b-4fcb-992e-85d116cf7e15-metrics-certs podName:73182ce3-cb4b-4fcb-992e-85d116cf7e15 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:26.892562284 +0000 UTC m=+18.136547934 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73182ce3-cb4b-4fcb-992e-85d116cf7e15-metrics-certs") pod "network-metrics-daemon-brx7q" (UID: "73182ce3-cb4b-4fcb-992e-85d116cf7e15") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:40:19.326104 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:19.325993 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:19.326235 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:19.326102 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlrnk" podUID="a1b523f4-d191-49bb-8e0e-4999e6fa1a92" Sep 11 14:40:19.326458 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:19.326442 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:40:19.326576 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:19.326555 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hp9w" podUID="ecdbcfa3-6309-46be-a9c9-ea8eb658de34" Sep 11 14:40:19.326728 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:19.326700 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:19.326829 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:19.326812 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brx7q" podUID="73182ce3-cb4b-4fcb-992e-85d116cf7e15" Sep 11 14:40:21.325013 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:21.324978 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:40:21.325013 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:21.325011 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:21.325511 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:21.324978 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:21.325511 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:21.325107 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hp9w" podUID="ecdbcfa3-6309-46be-a9c9-ea8eb658de34" Sep 11 14:40:21.325511 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:21.325195 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlrnk" podUID="a1b523f4-d191-49bb-8e0e-4999e6fa1a92" Sep 11 14:40:21.325511 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:21.325245 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brx7q" podUID="73182ce3-cb4b-4fcb-992e-85d116cf7e15" Sep 11 14:40:23.325101 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:23.325068 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:40:23.325504 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:23.325076 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:23.325504 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:23.325187 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hp9w" podUID="ecdbcfa3-6309-46be-a9c9-ea8eb658de34" Sep 11 14:40:23.325504 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:23.325283 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brx7q" podUID="73182ce3-cb4b-4fcb-992e-85d116cf7e15" Sep 11 14:40:23.325504 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:23.325319 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:23.325504 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:23.325397 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlrnk" podUID="a1b523f4-d191-49bb-8e0e-4999e6fa1a92" Sep 11 14:40:25.324316 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:25.324279 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:40:25.324316 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:25.324316 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:25.324864 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:25.324279 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:25.324864 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:25.324473 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hp9w" podUID="ecdbcfa3-6309-46be-a9c9-ea8eb658de34" Sep 11 14:40:25.324864 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:25.324594 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brx7q" podUID="73182ce3-cb4b-4fcb-992e-85d116cf7e15" Sep 11 14:40:25.324864 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:25.324670 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlrnk" podUID="a1b523f4-d191-49bb-8e0e-4999e6fa1a92" Sep 11 14:40:26.955133 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:26.955096 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8g5r\" (UniqueName: \"kubernetes.io/projected/ecdbcfa3-6309-46be-a9c9-ea8eb658de34-kube-api-access-p8g5r\") pod \"network-check-target-5hp9w\" (UID: \"ecdbcfa3-6309-46be-a9c9-ea8eb658de34\") " pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:40:26.955593 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:26.955161 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73182ce3-cb4b-4fcb-992e-85d116cf7e15-metrics-certs\") pod \"network-metrics-daemon-brx7q\" (UID: \"73182ce3-cb4b-4fcb-992e-85d116cf7e15\") " pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:26.955593 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:26.955213 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-original-pull-secret\") pod \"global-pull-secret-syncer-mlrnk\" (UID: \"a1b523f4-d191-49bb-8e0e-4999e6fa1a92\") " pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:26.955593 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:26.955307 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:40:26.955593 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:26.955323 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:40:26.955593 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:26.955338 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:40:26.955593 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:26.955353 2583 projected.go:196] Error preparing data for projected volume kube-api-access-p8g5r for pod openshift-network-diagnostics/network-check-target-5hp9w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:40:26.955593 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:26.955383 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73182ce3-cb4b-4fcb-992e-85d116cf7e15-metrics-certs podName:73182ce3-cb4b-4fcb-992e-85d116cf7e15 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:42.955369455 +0000 UTC m=+34.199355096 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73182ce3-cb4b-4fcb-992e-85d116cf7e15-metrics-certs") pod "network-metrics-daemon-brx7q" (UID: "73182ce3-cb4b-4fcb-992e-85d116cf7e15") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:40:26.955593 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:26.955405 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecdbcfa3-6309-46be-a9c9-ea8eb658de34-kube-api-access-p8g5r podName:ecdbcfa3-6309-46be-a9c9-ea8eb658de34 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:42.955389416 +0000 UTC m=+34.199375073 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-p8g5r" (UniqueName: "kubernetes.io/projected/ecdbcfa3-6309-46be-a9c9-ea8eb658de34-kube-api-access-p8g5r") pod "network-check-target-5hp9w" (UID: "ecdbcfa3-6309-46be-a9c9-ea8eb658de34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:40:26.955593 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:26.955323 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:40:26.955593 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:26.955458 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-original-pull-secret podName:a1b523f4-d191-49bb-8e0e-4999e6fa1a92 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:42.95544569 +0000 UTC m=+34.199431328 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-original-pull-secret") pod "global-pull-secret-syncer-mlrnk" (UID: "a1b523f4-d191-49bb-8e0e-4999e6fa1a92") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:40:27.324796 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:27.324697 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:40:27.324986 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:27.324730 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:27.324986 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:27.324836 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hp9w" podUID="ecdbcfa3-6309-46be-a9c9-ea8eb658de34" Sep 11 14:40:27.324986 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:27.324730 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:27.324986 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:27.324921 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brx7q" podUID="73182ce3-cb4b-4fcb-992e-85d116cf7e15" Sep 11 14:40:27.324986 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:27.324980 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlrnk" podUID="a1b523f4-d191-49bb-8e0e-4999e6fa1a92" Sep 11 14:40:29.325505 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:29.325462 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:29.325952 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:29.325573 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlrnk" podUID="a1b523f4-d191-49bb-8e0e-4999e6fa1a92" Sep 11 14:40:29.325952 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:29.325617 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:40:29.325952 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:29.325735 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hp9w" podUID="ecdbcfa3-6309-46be-a9c9-ea8eb658de34" Sep 11 14:40:29.325952 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:29.325759 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:29.325952 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:29.325874 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brx7q" podUID="73182ce3-cb4b-4fcb-992e-85d116cf7e15" Sep 11 14:40:30.475407 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:30.475058 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vfmp4" event={"ID":"ac306f32-44e2-4e36-8e09-2bb71e246eb8","Type":"ContainerStarted","Data":"0f1fa6186785b9d78247356cac7e446a1699b1a1299c1b3e5c491542f33dcff6"} Sep 11 14:40:30.476753 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:30.476722 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" event={"ID":"804431e5-f91c-4e79-911e-6dcadceeb6ef","Type":"ContainerStarted","Data":"533fd85cc9ce1f8e250f38931ba6117e18f1957084d4fab48bf3dca3c596c00e"} Sep 11 14:40:30.478109 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:30.478084 2583 generic.go:356] "Generic (PLEG): container finished" podID="c0b0394a-1326-4465-b674-23c0c6fbe017" containerID="c8a5059eabaa955aaa37ba9ba740bbdd775d064f6fa253796c538d11c84dc75c" exitCode=0 Sep 11 14:40:30.478210 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:30.478149 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75zzz" event={"ID":"c0b0394a-1326-4465-b674-23c0c6fbe017","Type":"ContainerDied","Data":"c8a5059eabaa955aaa37ba9ba740bbdd775d064f6fa253796c538d11c84dc75c"} Sep 11 14:40:30.481348 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:30.480094 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-97b8j" event={"ID":"d39f63c1-32f1-4c2a-8585-a3defd284630","Type":"ContainerStarted","Data":"1ef8d887ed107b6c6e2e1a28c53fe1b5843c8eca5937c2cb820302433b8d4da3"} Sep 11 14:40:30.484196 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:30.484171 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" event={"ID":"40743b05-b892-4d5a-bd13-46fb3822ad56","Type":"ContainerStarted","Data":"3ff38ba1daba931d3607978c7676cefe2d8f6040ece41292095742af91104db3"} Sep 11 14:40:30.484290 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:30.484242 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" event={"ID":"40743b05-b892-4d5a-bd13-46fb3822ad56","Type":"ContainerStarted","Data":"d25799c8684c2a76efe1bdd28feac969b426a60ab8ccfd8b5fced9ce0e382010"} Sep 11 14:40:30.484290 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:30.484279 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" event={"ID":"40743b05-b892-4d5a-bd13-46fb3822ad56","Type":"ContainerStarted","Data":"3fc81011d4b041b23799e727c8de02a4599b431a0aeba9711a6065ded43d0010"} Sep 11 14:40:30.484369 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:30.484315 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" event={"ID":"40743b05-b892-4d5a-bd13-46fb3822ad56","Type":"ContainerStarted","Data":"39d9bf0966f9fde92ae0007fb09ab2a6aa1f6627e1e576df04cfa9723e33c566"} Sep 11 14:40:30.484369 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:30.484359 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" event={"ID":"40743b05-b892-4d5a-bd13-46fb3822ad56","Type":"ContainerStarted","Data":"e77ec6567fb7b7557b2e9057525d6b31eabcaca9b9c96a17568922ac8d3cd0fa"} Sep 11 14:40:30.484436 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:30.484393 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" event={"ID":"40743b05-b892-4d5a-bd13-46fb3822ad56","Type":"ContainerStarted","Data":"33610cb66942862d9952959efdaf684505bbfd81e26a224e12a9ae12959f30dc"} Sep 11 14:40:30.485496 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:30.485475 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ppnhx" event={"ID":"fa59047c-1d81-44d0-b3a3-9e56bf53511c","Type":"ContainerStarted","Data":"501d90f57cb502ffaed2ea78159dd4f62066412ad5356433ad4c4cb50f6668e6"} Sep 11 14:40:30.487168 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:30.487142 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" event={"ID":"7b82ed19-be9c-4a91-bc2d-b1ff936194da","Type":"ContainerStarted","Data":"191ae2d660704536cb5f7f695abe92b7fd2e230c4413af097ef58af3ea03da6d"} Sep 11 14:40:30.488422 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:30.488402 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4vlml" event={"ID":"8dc4e60b-0baa-45d8-93b2-63436c43e3be","Type":"ContainerStarted","Data":"827a008a317ccebf66c613fdc7dbdf81ffd133d295e25b9349680175ac9685b5"} Sep 11 14:40:30.493513 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:30.491474 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-39.ec2.internal" podStartSLOduration=19.491441451 podStartE2EDuration="19.491441451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:14.46287669 +0000 UTC m=+5.706862352" watchObservedRunningTime="2026-09-11 14:40:30.491441451 +0000 UTC m=+21.735427111" Sep 11 14:40:30.493513 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:30.492758 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vfmp4" podStartSLOduration=4.424956717 podStartE2EDuration="21.492741238s" totalImagesPullingTime="17.067784521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:30.491413224 +0000 UTC m=+21.735398885" watchObservedRunningTime="2026-09-11 14:40:30.492741238 +0000 UTC m=+21.736726900" Sep 11 14:40:30.508972 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:30.508917 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7mbt2" podStartSLOduration=4.434206782 podStartE2EDuration="21.508901255s" totalImagesPullingTime="17.074694473s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:30.508741066 +0000 UTC m=+21.752726717" watchObservedRunningTime="2026-09-11 14:40:30.508901255 +0000 UTC m=+21.752886912" Sep 11 14:40:30.547426 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:30.547378 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4vlml" podStartSLOduration=4.478638425 podStartE2EDuration="21.547358998s" totalImagesPullingTime="17.068720573s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:30.547154003 +0000 UTC m=+21.791139661" watchObservedRunningTime="2026-09-11 14:40:30.547358998 +0000 UTC m=+21.791344657" Sep 11 14:40:30.562479 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:30.562429 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ppnhx" podStartSLOduration=3.343191994 podStartE2EDuration="21.562416132s" totalImagesPullingTime="18.219224138s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:30.56217642 +0000 UTC m=+21.806162078" watchObservedRunningTime="2026-09-11 14:40:30.562416132 +0000 UTC m=+21.806401789" Sep 11 14:40:30.580393 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:30.580343 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-97b8j" podStartSLOduration=4.424124126 podStartE2EDuration="21.580328382s" totalImagesPullingTime="17.156204256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:30.580161864 +0000 UTC m=+21.824147530" watchObservedRunningTime="2026-09-11 14:40:30.580328382 +0000 UTC m=+21.824314039" Sep 11 14:40:30.839683 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:30.839467 2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 11 14:40:31.181211 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:31.181097 2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T14:40:30.839680115Z","UUID":"d1ceed3d-2591-43e6-9efb-e2e24d1d8f18","Handler":null,"Name":"","Endpoint":""} Sep 11 14:40:31.182735 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:31.182698 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 11 14:40:31.182862 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:31.182748 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 11 14:40:31.324194 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:31.324115 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:40:31.324353 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:31.324115 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:31.324353 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:31.324241 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hp9w" podUID="ecdbcfa3-6309-46be-a9c9-ea8eb658de34" Sep 11 14:40:31.324353 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:31.324320 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlrnk" podUID="a1b523f4-d191-49bb-8e0e-4999e6fa1a92" Sep 11 14:40:31.324353 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:31.324120 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:31.324553 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:31.324416 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brx7q" podUID="73182ce3-cb4b-4fcb-992e-85d116cf7e15" Sep 11 14:40:31.492385 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:31.492350 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m7tfj" event={"ID":"f1b03dc4-474b-44b4-ba75-5feb9a72ebd3","Type":"ContainerStarted","Data":"d005ec59fce043333fd34558031ef70ddbb6beb754db3c4855e607baa3c16f9b"} Sep 11 14:40:31.494241 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:31.494214 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" event={"ID":"7b82ed19-be9c-4a91-bc2d-b1ff936194da","Type":"ContainerStarted","Data":"270c963b7540fbc139b4c33988d1953deb120a772bb1ddda66e700d906a51f83"} Sep 11 14:40:31.510369 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:31.510322 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-m7tfj" podStartSLOduration=4.280215694 podStartE2EDuration="22.510305501s" totalImagesPullingTime="18.230089807s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:31.50987507 +0000 UTC m=+22.753860728" watchObservedRunningTime="2026-09-11 14:40:31.510305501 +0000 UTC m=+22.754291160" Sep 11 14:40:32.388361 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:32.388332 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vfmp4" Sep 11 14:40:32.388974 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:32.388958 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vfmp4" Sep 11 14:40:32.501074 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:32.501045 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" event={"ID":"40743b05-b892-4d5a-bd13-46fb3822ad56","Type":"ContainerStarted","Data":"da634a70dbac9953df6053c634ab24abf7b647ea16ae25b7174d17be2f2729da"} Sep 11 14:40:32.503575 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:32.503544 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" event={"ID":"7b82ed19-be9c-4a91-bc2d-b1ff936194da","Type":"ContainerStarted","Data":"51d59cfdee3a14ef36d4334fecc3234c5bfe6d8d35459ce3c7af4c17668df362"} Sep 11 14:40:32.504206 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:32.504175 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vfmp4" Sep 11 14:40:32.504618 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:32.504597 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vfmp4" Sep 11 14:40:32.522729 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:32.522660 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h7fcz" podStartSLOduration=2.936054413 podStartE2EDuration="23.522644328s" totalImagesPullingTime="20.586589915s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:40:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:32.52232782 +0000 UTC m=+23.766313498" watchObservedRunningTime="2026-09-11 14:40:32.522644328 +0000 UTC m=+23.766629984" Sep 11 14:40:33.324925 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:33.324888 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:33.325207 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:33.324888 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:33.325207 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:33.324994 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlrnk" podUID="a1b523f4-d191-49bb-8e0e-4999e6fa1a92" Sep 11 14:40:33.325207 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:33.324888 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:40:33.325207 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:33.325069 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brx7q" podUID="73182ce3-cb4b-4fcb-992e-85d116cf7e15" Sep 11 14:40:33.325207 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:33.325148 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hp9w" podUID="ecdbcfa3-6309-46be-a9c9-ea8eb658de34" Sep 11 14:40:35.325016 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:35.324987 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:35.325448 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:35.324995 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:35.325448 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:35.325114 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlrnk" podUID="a1b523f4-d191-49bb-8e0e-4999e6fa1a92" Sep 11 14:40:35.325448 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:35.324995 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:40:35.325448 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:35.325202 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brx7q" podUID="73182ce3-cb4b-4fcb-992e-85d116cf7e15" Sep 11 14:40:35.325448 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:35.325268 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hp9w" podUID="ecdbcfa3-6309-46be-a9c9-ea8eb658de34" Sep 11 14:40:36.515430 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:36.515160 2583 generic.go:356] "Generic (PLEG): container finished" podID="c0b0394a-1326-4465-b674-23c0c6fbe017" containerID="bf2f2916f31d8f9743cc0681112734b3526599255e81cc3b09d153dd83e0fa47" exitCode=0 Sep 11 14:40:36.516153 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:36.515243 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75zzz" event={"ID":"c0b0394a-1326-4465-b674-23c0c6fbe017","Type":"ContainerDied","Data":"bf2f2916f31d8f9743cc0681112734b3526599255e81cc3b09d153dd83e0fa47"} Sep 11 14:40:36.519300 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:36.519277 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" event={"ID":"40743b05-b892-4d5a-bd13-46fb3822ad56","Type":"ContainerStarted","Data":"59ec8ea1155c34f6ad9f0de963599a4ddc263ab921c3678842e84d056f6a7c74"} Sep 11 14:40:36.519507 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:36.519492 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:36.519620 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:36.519590 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:36.519721 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:36.519630 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:36.533990 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:36.533966 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:36.534089 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:36.534066 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:40:36.569915 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:36.569871 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" podStartSLOduration=10.400945912 podStartE2EDuration="27.569856523s" totalImagesPullingTime="17.168910611s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:36.568870788 +0000 UTC m=+27.812856445" watchObservedRunningTime="2026-09-11 14:40:36.569856523 +0000 UTC m=+27.813842179" Sep 11 14:40:37.325056 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:37.325024 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:40:37.325208 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:37.325074 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:37.325208 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:37.325024 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:37.325208 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:37.325142 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hp9w" podUID="ecdbcfa3-6309-46be-a9c9-ea8eb658de34" Sep 11 14:40:37.325208 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:37.325190 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brx7q" podUID="73182ce3-cb4b-4fcb-992e-85d116cf7e15" Sep 11 14:40:37.325387 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:37.325285 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlrnk" podUID="a1b523f4-d191-49bb-8e0e-4999e6fa1a92" Sep 11 14:40:37.524292 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:37.524266 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75zzz" event={"ID":"c0b0394a-1326-4465-b674-23c0c6fbe017","Type":"ContainerStarted","Data":"e18bf9419475391c9aa0613c4ec8af6506ae25022c3f4ec71e04fc810489160f"} Sep 11 14:40:37.948486 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:37.948233 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mlrnk"] Sep 11 14:40:37.948656 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:37.948627 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:37.948777 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:37.948752 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlrnk" podUID="a1b523f4-d191-49bb-8e0e-4999e6fa1a92" Sep 11 14:40:37.951353 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:37.951325 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5hp9w"] Sep 11 14:40:37.951505 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:37.951429 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:40:37.951558 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:37.951514 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hp9w" podUID="ecdbcfa3-6309-46be-a9c9-ea8eb658de34" Sep 11 14:40:37.952075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:37.952046 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-brx7q"] Sep 11 14:40:37.952159 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:37.952131 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:37.952214 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:37.952201 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brx7q" podUID="73182ce3-cb4b-4fcb-992e-85d116cf7e15" Sep 11 14:40:38.529126 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:38.529095 2583 generic.go:356] "Generic (PLEG): container finished" podID="c0b0394a-1326-4465-b674-23c0c6fbe017" containerID="e18bf9419475391c9aa0613c4ec8af6506ae25022c3f4ec71e04fc810489160f" exitCode=0 Sep 11 14:40:38.529617 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:38.529168 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75zzz" event={"ID":"c0b0394a-1326-4465-b674-23c0c6fbe017","Type":"ContainerDied","Data":"e18bf9419475391c9aa0613c4ec8af6506ae25022c3f4ec71e04fc810489160f"} Sep 11 14:40:39.325732 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:39.325644 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:39.325879 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:39.325768 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brx7q" podUID="73182ce3-cb4b-4fcb-992e-85d116cf7e15" Sep 11 14:40:39.533979 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:39.533946 2583 generic.go:356] "Generic (PLEG): container finished" podID="c0b0394a-1326-4465-b674-23c0c6fbe017" containerID="47b952e9856f2cd2f59399143144c0c6043eddb7fdbaffe4dffecc444a7ef155" exitCode=0 Sep 11 14:40:39.534474 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:39.534000 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75zzz" event={"ID":"c0b0394a-1326-4465-b674-23c0c6fbe017","Type":"ContainerDied","Data":"47b952e9856f2cd2f59399143144c0c6043eddb7fdbaffe4dffecc444a7ef155"} Sep 11 14:40:40.324825 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:40.324796 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:40.324825 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:40.324821 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:40:40.325002 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:40.324897 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlrnk" podUID="a1b523f4-d191-49bb-8e0e-4999e6fa1a92" Sep 11 14:40:40.325034 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:40.325001 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hp9w" podUID="ecdbcfa3-6309-46be-a9c9-ea8eb658de34" Sep 11 14:40:41.324415 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:41.324380 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:41.325003 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:41.324521 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brx7q" podUID="73182ce3-cb4b-4fcb-992e-85d116cf7e15" Sep 11 14:40:42.325041 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.325008 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:40:42.325041 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.325030 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:42.325503 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:42.325123 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hp9w" podUID="ecdbcfa3-6309-46be-a9c9-ea8eb658de34" Sep 11 14:40:42.325503 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:42.325244 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlrnk" podUID="a1b523f4-d191-49bb-8e0e-4999e6fa1a92" Sep 11 14:40:42.515962 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.515880 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-39.ec2.internal" event="NodeReady" Sep 11 14:40:42.516113 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.516036 2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 11 14:40:42.564532 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.564218 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7mrb4"] Sep 11 14:40:42.643866 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.643834 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ht6zm"] Sep 11 14:40:42.644028 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.643928 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7mrb4" Sep 11 14:40:42.646232 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.646205 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 11 14:40:42.646340 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.646261 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 11 14:40:42.646340 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.646272 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vx9kl\"" Sep 11 14:40:42.658452 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.658424 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7mrb4"] Sep 11 14:40:42.658530 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.658515 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ht6zm"] Sep 11 14:40:42.658586 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.658519 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ht6zm" Sep 11 14:40:42.662477 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.661963 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 11 14:40:42.662477 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.662341 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tl5m6\"" Sep 11 14:40:42.662677 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.662613 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 11 14:40:42.664563 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.662811 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 11 14:40:42.778049 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.777939 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70ec3434-0593-4acd-bd81-2580090a5020-metrics-tls\") pod \"dns-default-7mrb4\" (UID: \"70ec3434-0593-4acd-bd81-2580090a5020\") " pod="openshift-dns/dns-default-7mrb4" Sep 11 14:40:42.778200 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.778046 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/062d13b8-8971-4c91-b348-8b609ed42b56-cert\") pod \"ingress-canary-ht6zm\" (UID: \"062d13b8-8971-4c91-b348-8b609ed42b56\") " pod="openshift-ingress-canary/ingress-canary-ht6zm" Sep 11 14:40:42.778200 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.778080 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js7rm\" (UniqueName: \"kubernetes.io/projected/70ec3434-0593-4acd-bd81-2580090a5020-kube-api-access-js7rm\") pod \"dns-default-7mrb4\" (UID: \"70ec3434-0593-4acd-bd81-2580090a5020\") " pod="openshift-dns/dns-default-7mrb4" Sep 11 14:40:42.778200 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.778124 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/70ec3434-0593-4acd-bd81-2580090a5020-config-volume\") pod \"dns-default-7mrb4\" (UID: \"70ec3434-0593-4acd-bd81-2580090a5020\") " pod="openshift-dns/dns-default-7mrb4" Sep 11 14:40:42.778200 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.778170 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ec3434-0593-4acd-bd81-2580090a5020-tmp-dir\") pod \"dns-default-7mrb4\" (UID: \"70ec3434-0593-4acd-bd81-2580090a5020\") " pod="openshift-dns/dns-default-7mrb4" Sep 11 14:40:42.778372 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.778222 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nktkp\" (UniqueName: \"kubernetes.io/projected/062d13b8-8971-4c91-b348-8b609ed42b56-kube-api-access-nktkp\") pod \"ingress-canary-ht6zm\" (UID: \"062d13b8-8971-4c91-b348-8b609ed42b56\") " pod="openshift-ingress-canary/ingress-canary-ht6zm" Sep 11 14:40:42.878868 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.878829 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/70ec3434-0593-4acd-bd81-2580090a5020-config-volume\") pod \"dns-default-7mrb4\" (UID: \"70ec3434-0593-4acd-bd81-2580090a5020\") " pod="openshift-dns/dns-default-7mrb4" Sep 11 14:40:42.878868 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.878876 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ec3434-0593-4acd-bd81-2580090a5020-tmp-dir\") pod \"dns-default-7mrb4\" (UID: \"70ec3434-0593-4acd-bd81-2580090a5020\") " pod="openshift-dns/dns-default-7mrb4" Sep 11 14:40:42.879103 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.878909 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nktkp\" (UniqueName: \"kubernetes.io/projected/062d13b8-8971-4c91-b348-8b609ed42b56-kube-api-access-nktkp\") pod \"ingress-canary-ht6zm\" (UID: \"062d13b8-8971-4c91-b348-8b609ed42b56\") " pod="openshift-ingress-canary/ingress-canary-ht6zm" Sep 11 14:40:42.879103 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.878955 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70ec3434-0593-4acd-bd81-2580090a5020-metrics-tls\") pod \"dns-default-7mrb4\" (UID: \"70ec3434-0593-4acd-bd81-2580090a5020\") " pod="openshift-dns/dns-default-7mrb4" Sep 11 14:40:42.879103 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.879015 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/062d13b8-8971-4c91-b348-8b609ed42b56-cert\") pod \"ingress-canary-ht6zm\" (UID: \"062d13b8-8971-4c91-b348-8b609ed42b56\") " pod="openshift-ingress-canary/ingress-canary-ht6zm" Sep 11 14:40:42.879103 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.879039 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js7rm\" (UniqueName: \"kubernetes.io/projected/70ec3434-0593-4acd-bd81-2580090a5020-kube-api-access-js7rm\") pod \"dns-default-7mrb4\" (UID: \"70ec3434-0593-4acd-bd81-2580090a5020\") " pod="openshift-dns/dns-default-7mrb4" Sep 11 14:40:42.879298 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:42.879198 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:40:42.879298 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:42.879259 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70ec3434-0593-4acd-bd81-2580090a5020-metrics-tls podName:70ec3434-0593-4acd-bd81-2580090a5020 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:43.37924114 +0000 UTC m=+34.623226785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/70ec3434-0593-4acd-bd81-2580090a5020-metrics-tls") pod "dns-default-7mrb4" (UID: "70ec3434-0593-4acd-bd81-2580090a5020") : secret "dns-default-metrics-tls" not found Sep 11 14:40:42.879495 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.879468 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/70ec3434-0593-4acd-bd81-2580090a5020-config-volume\") pod \"dns-default-7mrb4\" (UID: \"70ec3434-0593-4acd-bd81-2580090a5020\") " pod="openshift-dns/dns-default-7mrb4" Sep 11 14:40:42.884523 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.884476 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/062d13b8-8971-4c91-b348-8b609ed42b56-cert\") pod \"ingress-canary-ht6zm\" (UID: \"062d13b8-8971-4c91-b348-8b609ed42b56\") " pod="openshift-ingress-canary/ingress-canary-ht6zm" Sep 11 14:40:42.887898 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.887821 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nktkp\" (UniqueName: \"kubernetes.io/projected/062d13b8-8971-4c91-b348-8b609ed42b56-kube-api-access-nktkp\") pod \"ingress-canary-ht6zm\" (UID: \"062d13b8-8971-4c91-b348-8b609ed42b56\") " pod="openshift-ingress-canary/ingress-canary-ht6zm" Sep 11 14:40:42.887999 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.887907 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js7rm\" (UniqueName: \"kubernetes.io/projected/70ec3434-0593-4acd-bd81-2580090a5020-kube-api-access-js7rm\") pod \"dns-default-7mrb4\" (UID: \"70ec3434-0593-4acd-bd81-2580090a5020\") " pod="openshift-dns/dns-default-7mrb4" Sep 11 14:40:42.889336 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.889314 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ec3434-0593-4acd-bd81-2580090a5020-tmp-dir\") pod \"dns-default-7mrb4\" (UID: \"70ec3434-0593-4acd-bd81-2580090a5020\") " pod="openshift-dns/dns-default-7mrb4" Sep 11 14:40:42.969740 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.969537 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ht6zm" Sep 11 14:40:42.979944 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.979917 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-original-pull-secret\") pod \"global-pull-secret-syncer-mlrnk\" (UID: \"a1b523f4-d191-49bb-8e0e-4999e6fa1a92\") " pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:42.980117 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.979976 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8g5r\" (UniqueName: \"kubernetes.io/projected/ecdbcfa3-6309-46be-a9c9-ea8eb658de34-kube-api-access-p8g5r\") pod \"network-check-target-5hp9w\" (UID: \"ecdbcfa3-6309-46be-a9c9-ea8eb658de34\") " pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:40:42.980117 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:42.980043 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73182ce3-cb4b-4fcb-992e-85d116cf7e15-metrics-certs\") pod \"network-metrics-daemon-brx7q\" (UID: \"73182ce3-cb4b-4fcb-992e-85d116cf7e15\") " pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:42.980117 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:42.980111 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:40:42.980317 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:42.980131 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:40:42.980317 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:42.980143 2583 projected.go:196] Error preparing data for projected volume kube-api-access-p8g5r for pod openshift-network-diagnostics/network-check-target-5hp9w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:40:42.980317 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:42.980161 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:40:42.980317 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:42.980199 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecdbcfa3-6309-46be-a9c9-ea8eb658de34-kube-api-access-p8g5r podName:ecdbcfa3-6309-46be-a9c9-ea8eb658de34 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:14.980182627 +0000 UTC m=+66.224168283 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-p8g5r" (UniqueName: "kubernetes.io/projected/ecdbcfa3-6309-46be-a9c9-ea8eb658de34-kube-api-access-p8g5r") pod "network-check-target-5hp9w" (UID: "ecdbcfa3-6309-46be-a9c9-ea8eb658de34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:40:42.980317 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:42.980230 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73182ce3-cb4b-4fcb-992e-85d116cf7e15-metrics-certs podName:73182ce3-cb4b-4fcb-992e-85d116cf7e15 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:14.980218414 +0000 UTC m=+66.224204051 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73182ce3-cb4b-4fcb-992e-85d116cf7e15-metrics-certs") pod "network-metrics-daemon-brx7q" (UID: "73182ce3-cb4b-4fcb-992e-85d116cf7e15") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:40:42.980317 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:42.980162 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:40:42.980317 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:42.980313 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-original-pull-secret podName:a1b523f4-d191-49bb-8e0e-4999e6fa1a92 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:14.980297331 +0000 UTC m=+66.224282975 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-original-pull-secret") pod "global-pull-secret-syncer-mlrnk" (UID: "a1b523f4-d191-49bb-8e0e-4999e6fa1a92") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:40:43.324319 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:43.324284 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:40:43.326883 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:43.326848 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 14:40:43.327262 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:43.326895 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jjd7d\"" Sep 11 14:40:43.383648 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:43.383619 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70ec3434-0593-4acd-bd81-2580090a5020-metrics-tls\") pod \"dns-default-7mrb4\" (UID: \"70ec3434-0593-4acd-bd81-2580090a5020\") " pod="openshift-dns/dns-default-7mrb4" Sep 11 14:40:43.383853 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:43.383826 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:40:43.383924 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:43.383910 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70ec3434-0593-4acd-bd81-2580090a5020-metrics-tls podName:70ec3434-0593-4acd-bd81-2580090a5020 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:44.383890863 +0000 UTC m=+35.627876500 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/70ec3434-0593-4acd-bd81-2580090a5020-metrics-tls") pod "dns-default-7mrb4" (UID: "70ec3434-0593-4acd-bd81-2580090a5020") : secret "dns-default-metrics-tls" not found Sep 11 14:40:44.324531 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:44.324454 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:40:44.324531 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:44.324495 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:40:44.327676 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:44.327655 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 14:40:44.328303 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:44.328279 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xdw2d\"" Sep 11 14:40:44.328396 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:44.328301 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 14:40:44.328396 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:44.328357 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 14:40:44.390833 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:44.390803 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70ec3434-0593-4acd-bd81-2580090a5020-metrics-tls\") pod \"dns-default-7mrb4\" (UID: \"70ec3434-0593-4acd-bd81-2580090a5020\") " pod="openshift-dns/dns-default-7mrb4" Sep 11 14:40:44.390996 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:44.390910 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:40:44.390996 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:44.390964 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70ec3434-0593-4acd-bd81-2580090a5020-metrics-tls podName:70ec3434-0593-4acd-bd81-2580090a5020 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:46.390950424 +0000 UTC m=+37.634936061 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/70ec3434-0593-4acd-bd81-2580090a5020-metrics-tls") pod "dns-default-7mrb4" (UID: "70ec3434-0593-4acd-bd81-2580090a5020") : secret "dns-default-metrics-tls" not found Sep 11 14:40:44.547491 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:44.547468 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ht6zm"] Sep 11 14:40:44.551976 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:40:44.551951 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod062d13b8_8971_4c91_b348_8b609ed42b56.slice/crio-9c4b888adf7f4b91ceca346bd415097c0265c8793fce42cc4b98de91a8556825 WatchSource:0}: Error finding container 9c4b888adf7f4b91ceca346bd415097c0265c8793fce42cc4b98de91a8556825: Status 404 returned error can't find the container with id 9c4b888adf7f4b91ceca346bd415097c0265c8793fce42cc4b98de91a8556825 Sep 11 14:40:45.551817 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:45.551782 2583 generic.go:356] "Generic (PLEG): container finished" podID="c0b0394a-1326-4465-b674-23c0c6fbe017" containerID="ab7c06ab20970ac6abeca20d805c5d67208c36dc739faceabb2d71612d690446" exitCode=0 Sep 11 14:40:45.552561 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:45.551862 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75zzz" event={"ID":"c0b0394a-1326-4465-b674-23c0c6fbe017","Type":"ContainerDied","Data":"ab7c06ab20970ac6abeca20d805c5d67208c36dc739faceabb2d71612d690446"} Sep 11 14:40:45.553235 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:45.553206 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ht6zm" event={"ID":"062d13b8-8971-4c91-b348-8b609ed42b56","Type":"ContainerStarted","Data":"9c4b888adf7f4b91ceca346bd415097c0265c8793fce42cc4b98de91a8556825"} Sep 11 14:40:46.406604 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:46.406561 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70ec3434-0593-4acd-bd81-2580090a5020-metrics-tls\") pod \"dns-default-7mrb4\" (UID: \"70ec3434-0593-4acd-bd81-2580090a5020\") " pod="openshift-dns/dns-default-7mrb4" Sep 11 14:40:46.406793 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:46.406695 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:40:46.406793 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:46.406773 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70ec3434-0593-4acd-bd81-2580090a5020-metrics-tls podName:70ec3434-0593-4acd-bd81-2580090a5020 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:50.406756106 +0000 UTC m=+41.650741745 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/70ec3434-0593-4acd-bd81-2580090a5020-metrics-tls") pod "dns-default-7mrb4" (UID: "70ec3434-0593-4acd-bd81-2580090a5020") : secret "dns-default-metrics-tls" not found Sep 11 14:40:46.559066 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:46.559028 2583 generic.go:356] "Generic (PLEG): container finished" podID="c0b0394a-1326-4465-b674-23c0c6fbe017" containerID="a44ad49596f5aef07aceb46b6890782310be3c7fc91258bdd1f666f2e09f1613" exitCode=0 Sep 11 14:40:46.559494 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:46.559096 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75zzz" event={"ID":"c0b0394a-1326-4465-b674-23c0c6fbe017","Type":"ContainerDied","Data":"a44ad49596f5aef07aceb46b6890782310be3c7fc91258bdd1f666f2e09f1613"} Sep 11 14:40:47.565442 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:47.565199 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75zzz" event={"ID":"c0b0394a-1326-4465-b674-23c0c6fbe017","Type":"ContainerStarted","Data":"9cbe6acded57c114f757038fb72c3a5f91fe4dc7a305ce708076e83fdb5d9062"} Sep 11 14:40:47.566773 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:47.566748 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ht6zm" event={"ID":"062d13b8-8971-4c91-b348-8b609ed42b56","Type":"ContainerStarted","Data":"b889f133a08827300d06e3a17c374ac25a176d93bae6b142cfe11af3d4b81115"} Sep 11 14:40:47.589114 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:47.589058 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-75zzz" podStartSLOduration=9.524057287 podStartE2EDuration="38.589038451s" totalImagesPullingTime="29.064981164s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:40:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:47.588488074 +0000 UTC m=+38.832473731" watchObservedRunningTime="2026-09-11 14:40:47.589038451 +0000 UTC m=+38.833024110" Sep 11 14:40:47.603782 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:47.603662 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ht6zm" podStartSLOduration=2.755305034 podStartE2EDuration="5.603642895s" totalImagesPullingTime="2.848337861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:47.603205557 +0000 UTC m=+38.847191215" watchObservedRunningTime="2026-09-11 14:40:47.603642895 +0000 UTC m=+38.847628548" Sep 11 14:40:50.435073 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:50.435032 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70ec3434-0593-4acd-bd81-2580090a5020-metrics-tls\") pod \"dns-default-7mrb4\" (UID: \"70ec3434-0593-4acd-bd81-2580090a5020\") " pod="openshift-dns/dns-default-7mrb4" Sep 11 14:40:50.435472 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:50.435170 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:40:50.435472 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:50.435230 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70ec3434-0593-4acd-bd81-2580090a5020-metrics-tls podName:70ec3434-0593-4acd-bd81-2580090a5020 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:58.435215262 +0000 UTC m=+49.679200898 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/70ec3434-0593-4acd-bd81-2580090a5020-metrics-tls") pod "dns-default-7mrb4" (UID: "70ec3434-0593-4acd-bd81-2580090a5020") : secret "dns-default-metrics-tls" not found Sep 11 14:40:55.961626 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:55.961593 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-lwrt2"] Sep 11 14:40:55.991208 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:55.991178 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-lwrt2"] Sep 11 14:40:55.991350 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:55.991271 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-lwrt2" Sep 11 14:40:55.993671 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:55.993647 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 11 14:40:55.993795 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:55.993763 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-q6x5j\"" Sep 11 14:40:55.994629 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:55.994610 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 11 14:40:55.994698 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:55.994668 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 11 14:40:55.995567 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:55.995548 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 11 14:40:55.998890 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:55.998874 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 11 14:40:56.174547 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.174511 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/57d4ba26-384d-42f8-86a1-335e27cdf762-config\") pod \"console-operator-696884fb9c-lwrt2\" (UID: \"57d4ba26-384d-42f8-86a1-335e27cdf762\") " pod="openshift-console-operator/console-operator-696884fb9c-lwrt2" Sep 11 14:40:56.174547 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.174551 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h48w7\" (UniqueName: \"kubernetes.io/projected/57d4ba26-384d-42f8-86a1-335e27cdf762-kube-api-access-h48w7\") pod \"console-operator-696884fb9c-lwrt2\" (UID: \"57d4ba26-384d-42f8-86a1-335e27cdf762\") " pod="openshift-console-operator/console-operator-696884fb9c-lwrt2" Sep 11 14:40:56.174798 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.174643 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57d4ba26-384d-42f8-86a1-335e27cdf762-trusted-ca\") pod \"console-operator-696884fb9c-lwrt2\" (UID: \"57d4ba26-384d-42f8-86a1-335e27cdf762\") " pod="openshift-console-operator/console-operator-696884fb9c-lwrt2" Sep 11 14:40:56.174798 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.174683 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/57d4ba26-384d-42f8-86a1-335e27cdf762-serving-cert\") pod \"console-operator-696884fb9c-lwrt2\" (UID: \"57d4ba26-384d-42f8-86a1-335e27cdf762\") " pod="openshift-console-operator/console-operator-696884fb9c-lwrt2" Sep 11 14:40:56.275872 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.275778 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/57d4ba26-384d-42f8-86a1-335e27cdf762-serving-cert\") pod \"console-operator-696884fb9c-lwrt2\" (UID: \"57d4ba26-384d-42f8-86a1-335e27cdf762\") " pod="openshift-console-operator/console-operator-696884fb9c-lwrt2" Sep 11 14:40:56.275872 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.275861 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/57d4ba26-384d-42f8-86a1-335e27cdf762-config\") pod \"console-operator-696884fb9c-lwrt2\" (UID: \"57d4ba26-384d-42f8-86a1-335e27cdf762\") " pod="openshift-console-operator/console-operator-696884fb9c-lwrt2" Sep 11 14:40:56.276096 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.275880 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h48w7\" (UniqueName: \"kubernetes.io/projected/57d4ba26-384d-42f8-86a1-335e27cdf762-kube-api-access-h48w7\") pod \"console-operator-696884fb9c-lwrt2\" (UID: \"57d4ba26-384d-42f8-86a1-335e27cdf762\") " pod="openshift-console-operator/console-operator-696884fb9c-lwrt2" Sep 11 14:40:56.276096 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.275928 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57d4ba26-384d-42f8-86a1-335e27cdf762-trusted-ca\") pod \"console-operator-696884fb9c-lwrt2\" (UID: \"57d4ba26-384d-42f8-86a1-335e27cdf762\") " pod="openshift-console-operator/console-operator-696884fb9c-lwrt2" Sep 11 14:40:56.276634 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.276615 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/57d4ba26-384d-42f8-86a1-335e27cdf762-config\") pod \"console-operator-696884fb9c-lwrt2\" (UID: \"57d4ba26-384d-42f8-86a1-335e27cdf762\") " pod="openshift-console-operator/console-operator-696884fb9c-lwrt2" Sep 11 14:40:56.276701 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.276685 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57d4ba26-384d-42f8-86a1-335e27cdf762-trusted-ca\") pod \"console-operator-696884fb9c-lwrt2\" (UID: \"57d4ba26-384d-42f8-86a1-335e27cdf762\") " pod="openshift-console-operator/console-operator-696884fb9c-lwrt2" Sep 11 14:40:56.279207 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.279191 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/57d4ba26-384d-42f8-86a1-335e27cdf762-serving-cert\") pod \"console-operator-696884fb9c-lwrt2\" (UID: \"57d4ba26-384d-42f8-86a1-335e27cdf762\") " pod="openshift-console-operator/console-operator-696884fb9c-lwrt2" Sep 11 14:40:56.285154 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.285136 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h48w7\" (UniqueName: \"kubernetes.io/projected/57d4ba26-384d-42f8-86a1-335e27cdf762-kube-api-access-h48w7\") pod \"console-operator-696884fb9c-lwrt2\" (UID: \"57d4ba26-384d-42f8-86a1-335e27cdf762\") " pod="openshift-console-operator/console-operator-696884fb9c-lwrt2" Sep 11 14:40:56.301191 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.301171 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-lwrt2" Sep 11 14:40:56.474035 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.474003 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56f4498d5f-4dwl8"] Sep 11 14:40:56.497949 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.497923 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56f4498d5f-4dwl8"] Sep 11 14:40:56.498090 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.498020 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:40:56.501745 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.501718 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 11 14:40:56.501857 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.501759 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6xrxr\"" Sep 11 14:40:56.502085 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.502067 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 11 14:40:56.502691 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.502674 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 11 14:40:56.507650 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.507631 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 11 14:40:56.596185 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.596107 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-lwrt2"] Sep 11 14:40:56.680236 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.680199 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6f478f4-6ae4-4ed8-8881-b1945228d966-image-registry-private-configuration\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:40:56.680383 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.680251 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6f478f4-6ae4-4ed8-8881-b1945228d966-trusted-ca\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:40:56.680383 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.680275 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-bound-sa-token\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:40:56.680383 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.680326 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-registry-tls\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:40:56.680383 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.680355 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6f478f4-6ae4-4ed8-8881-b1945228d966-installation-pull-secrets\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:40:56.680525 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.680394 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6f478f4-6ae4-4ed8-8881-b1945228d966-registry-certificates\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:40:56.680525 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.680415 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmg2g\" (UniqueName: \"kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-kube-api-access-jmg2g\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:40:56.680525 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.680460 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6f478f4-6ae4-4ed8-8881-b1945228d966-ca-trust-extracted\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:40:56.781665 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.781634 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6f478f4-6ae4-4ed8-8881-b1945228d966-registry-certificates\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:40:56.781665 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.781667 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmg2g\" (UniqueName: \"kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-kube-api-access-jmg2g\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:40:56.781848 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.781689 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6f478f4-6ae4-4ed8-8881-b1945228d966-ca-trust-extracted\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:40:56.781895 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.781841 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6f478f4-6ae4-4ed8-8881-b1945228d966-image-registry-private-configuration\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:40:56.781945 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.781914 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6f478f4-6ae4-4ed8-8881-b1945228d966-trusted-ca\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:40:56.781998 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.781940 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-bound-sa-token\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:40:56.781998 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.781978 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-registry-tls\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:40:56.782087 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.782002 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6f478f4-6ae4-4ed8-8881-b1945228d966-installation-pull-secrets\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:40:56.782136 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:56.782095 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 14:40:56.782136 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:56.782123 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56f4498d5f-4dwl8: secret "image-registry-tls" not found Sep 11 14:40:56.782313 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.782137 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6f478f4-6ae4-4ed8-8881-b1945228d966-ca-trust-extracted\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:40:56.782313 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:56.782175 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-registry-tls podName:c6f478f4-6ae4-4ed8-8881-b1945228d966 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:57.282157958 +0000 UTC m=+48.526143613 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-registry-tls") pod "image-registry-56f4498d5f-4dwl8" (UID: "c6f478f4-6ae4-4ed8-8881-b1945228d966") : secret "image-registry-tls" not found Sep 11 14:40:56.782445 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.782331 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6f478f4-6ae4-4ed8-8881-b1945228d966-registry-certificates\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:40:56.782891 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.782874 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6f478f4-6ae4-4ed8-8881-b1945228d966-trusted-ca\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:40:56.784236 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.784214 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6f478f4-6ae4-4ed8-8881-b1945228d966-image-registry-private-configuration\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:40:56.784326 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.784306 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6f478f4-6ae4-4ed8-8881-b1945228d966-installation-pull-secrets\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:40:56.791185 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.791165 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-bound-sa-token\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:40:56.791417 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:56.791400 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmg2g\" (UniqueName: \"kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-kube-api-access-jmg2g\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:40:57.285927 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:57.285888 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-registry-tls\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:40:57.286481 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:57.286352 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 14:40:57.286481 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:57.286382 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56f4498d5f-4dwl8: secret "image-registry-tls" not found Sep 11 14:40:57.286481 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:57.286460 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-registry-tls podName:c6f478f4-6ae4-4ed8-8881-b1945228d966 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:58.286439101 +0000 UTC m=+49.530424737 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-registry-tls") pod "image-registry-56f4498d5f-4dwl8" (UID: "c6f478f4-6ae4-4ed8-8881-b1945228d966") : secret "image-registry-tls" not found Sep 11 14:40:57.590734 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:57.590633 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-lwrt2" event={"ID":"57d4ba26-384d-42f8-86a1-335e27cdf762","Type":"ContainerStarted","Data":"b94909c41d0af015cc1b9e4568107c28eb6d3072a11ded63fab270dfe3b7064c"} Sep 11 14:40:58.295316 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:58.295277 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-registry-tls\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:40:58.295827 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:58.295435 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 14:40:58.295827 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:58.295454 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56f4498d5f-4dwl8: secret "image-registry-tls" not found Sep 11 14:40:58.295827 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:58.295514 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-registry-tls podName:c6f478f4-6ae4-4ed8-8881-b1945228d966 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:00.295498422 +0000 UTC m=+51.539484059 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-registry-tls") pod "image-registry-56f4498d5f-4dwl8" (UID: "c6f478f4-6ae4-4ed8-8881-b1945228d966") : secret "image-registry-tls" not found Sep 11 14:40:58.496664 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:58.496620 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70ec3434-0593-4acd-bd81-2580090a5020-metrics-tls\") pod \"dns-default-7mrb4\" (UID: \"70ec3434-0593-4acd-bd81-2580090a5020\") " pod="openshift-dns/dns-default-7mrb4" Sep 11 14:40:58.496852 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:58.496792 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:40:58.496901 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:40:58.496888 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70ec3434-0593-4acd-bd81-2580090a5020-metrics-tls podName:70ec3434-0593-4acd-bd81-2580090a5020 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:14.496870224 +0000 UTC m=+65.740855860 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/70ec3434-0593-4acd-bd81-2580090a5020-metrics-tls") pod "dns-default-7mrb4" (UID: "70ec3434-0593-4acd-bd81-2580090a5020") : secret "dns-default-metrics-tls" not found Sep 11 14:40:59.597452 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:59.597381 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lwrt2_57d4ba26-384d-42f8-86a1-335e27cdf762/console-operator/0.log" Sep 11 14:40:59.597452 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:59.597421 2583 generic.go:356] "Generic (PLEG): container finished" podID="57d4ba26-384d-42f8-86a1-335e27cdf762" containerID="9123fee6fe37c25b6ec34e98b067948b38868a04ecaa3ae8cdd825d2b2e64e79" exitCode=255 Sep 11 14:40:59.597827 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:59.597453 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-lwrt2" event={"ID":"57d4ba26-384d-42f8-86a1-335e27cdf762","Type":"ContainerDied","Data":"9123fee6fe37c25b6ec34e98b067948b38868a04ecaa3ae8cdd825d2b2e64e79"} Sep 11 14:40:59.597827 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:40:59.597727 2583 scope.go:117] "RemoveContainer" containerID="9123fee6fe37c25b6ec34e98b067948b38868a04ecaa3ae8cdd825d2b2e64e79" Sep 11 14:41:00.310900 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:00.310862 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-registry-tls\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:41:00.311082 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:00.311016 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 14:41:00.311082 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:00.311039 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56f4498d5f-4dwl8: secret "image-registry-tls" not found Sep 11 14:41:00.311167 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:00.311095 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-registry-tls podName:c6f478f4-6ae4-4ed8-8881-b1945228d966 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:04.311078119 +0000 UTC m=+55.555063756 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-registry-tls") pod "image-registry-56f4498d5f-4dwl8" (UID: "c6f478f4-6ae4-4ed8-8881-b1945228d966") : secret "image-registry-tls" not found Sep 11 14:41:00.601659 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:00.601591 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lwrt2_57d4ba26-384d-42f8-86a1-335e27cdf762/console-operator/1.log" Sep 11 14:41:00.602013 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:00.601956 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lwrt2_57d4ba26-384d-42f8-86a1-335e27cdf762/console-operator/0.log" Sep 11 14:41:00.602013 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:00.601985 2583 generic.go:356] "Generic (PLEG): container finished" podID="57d4ba26-384d-42f8-86a1-335e27cdf762" containerID="ab577f6246421649fbb9e8f5ce62f47ff0856948fc9ce8bac27eca8a24def6d3" exitCode=255 Sep 11 14:41:00.602087 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:00.602034 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-lwrt2" event={"ID":"57d4ba26-384d-42f8-86a1-335e27cdf762","Type":"ContainerDied","Data":"ab577f6246421649fbb9e8f5ce62f47ff0856948fc9ce8bac27eca8a24def6d3"} Sep 11 14:41:00.602122 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:00.602087 2583 scope.go:117] "RemoveContainer" containerID="9123fee6fe37c25b6ec34e98b067948b38868a04ecaa3ae8cdd825d2b2e64e79" Sep 11 14:41:00.602295 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:00.602273 2583 scope.go:117] "RemoveContainer" containerID="ab577f6246421649fbb9e8f5ce62f47ff0856948fc9ce8bac27eca8a24def6d3" Sep 11 14:41:00.602470 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:00.602453 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-lwrt2_openshift-console-operator(57d4ba26-384d-42f8-86a1-335e27cdf762)\"" pod="openshift-console-operator/console-operator-696884fb9c-lwrt2" podUID="57d4ba26-384d-42f8-86a1-335e27cdf762" Sep 11 14:41:00.897327 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:00.897296 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-th99j"] Sep 11 14:41:00.899670 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:00.899649 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-th99j" Sep 11 14:41:00.915680 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:00.915660 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 11 14:41:00.915775 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:00.915683 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 11 14:41:00.915775 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:00.915753 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 11 14:41:00.916083 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:00.916070 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 11 14:41:00.916431 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:00.916418 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6tvzn\"" Sep 11 14:41:00.957021 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:00.956990 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-th99j"] Sep 11 14:41:01.015169 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:01.015134 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-crio-socket\") pod \"insights-runtime-extractor-th99j\" (UID: \"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b\") " pod="openshift-insights/insights-runtime-extractor-th99j" Sep 11 14:41:01.015169 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:01.015172 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-th99j\" (UID: \"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b\") " pod="openshift-insights/insights-runtime-extractor-th99j" Sep 11 14:41:01.015358 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:01.015284 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-th99j\" (UID: \"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b\") " pod="openshift-insights/insights-runtime-extractor-th99j" Sep 11 14:41:01.015358 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:01.015317 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-data-volume\") pod \"insights-runtime-extractor-th99j\" (UID: \"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b\") " pod="openshift-insights/insights-runtime-extractor-th99j" Sep 11 14:41:01.015358 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:01.015339 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzr8h\" (UniqueName: \"kubernetes.io/projected/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-kube-api-access-jzr8h\") pod \"insights-runtime-extractor-th99j\" (UID: \"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b\") " pod="openshift-insights/insights-runtime-extractor-th99j" Sep 11 14:41:01.115979 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:01.115949 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-th99j\" (UID: \"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b\") " pod="openshift-insights/insights-runtime-extractor-th99j" Sep 11 14:41:01.115979 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:01.115985 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-data-volume\") pod \"insights-runtime-extractor-th99j\" (UID: \"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b\") " pod="openshift-insights/insights-runtime-extractor-th99j" Sep 11 14:41:01.116166 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:01.116019 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzr8h\" (UniqueName: \"kubernetes.io/projected/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-kube-api-access-jzr8h\") pod \"insights-runtime-extractor-th99j\" (UID: \"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b\") " pod="openshift-insights/insights-runtime-extractor-th99j" Sep 11 14:41:01.116166 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:01.116071 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 14:41:01.116166 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:01.116083 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-crio-socket\") pod \"insights-runtime-extractor-th99j\" (UID: \"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b\") " pod="openshift-insights/insights-runtime-extractor-th99j" Sep 11 14:41:01.116166 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:01.116105 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-th99j\" (UID: \"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b\") " pod="openshift-insights/insights-runtime-extractor-th99j" Sep 11 14:41:01.116166 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:01.116156 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-kube-rbac-proxy-cm podName:0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b nodeName:}" failed. No retries permitted until 2026-09-11 14:41:01.61613725 +0000 UTC m=+52.860122902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-th99j" (UID: "0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b") : configmap "kube-rbac-proxy" not found Sep 11 14:41:01.116356 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:01.116178 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-crio-socket\") pod \"insights-runtime-extractor-th99j\" (UID: \"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b\") " pod="openshift-insights/insights-runtime-extractor-th99j" Sep 11 14:41:01.116356 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:01.116212 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:41:01.116356 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:01.116260 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-insights-runtime-extractor-tls podName:0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b nodeName:}" failed. No retries permitted until 2026-09-11 14:41:01.616251767 +0000 UTC m=+52.860237404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-th99j" (UID: "0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b") : secret "insights-runtime-extractor-tls" not found Sep 11 14:41:01.116356 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:01.116337 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-data-volume\") pod \"insights-runtime-extractor-th99j\" (UID: \"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b\") " pod="openshift-insights/insights-runtime-extractor-th99j" Sep 11 14:41:01.125007 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:01.124982 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzr8h\" (UniqueName: \"kubernetes.io/projected/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-kube-api-access-jzr8h\") pod \"insights-runtime-extractor-th99j\" (UID: \"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b\") " pod="openshift-insights/insights-runtime-extractor-th99j" Sep 11 14:41:01.605989 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:01.605963 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lwrt2_57d4ba26-384d-42f8-86a1-335e27cdf762/console-operator/1.log" Sep 11 14:41:01.606405 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:01.606260 2583 scope.go:117] "RemoveContainer" containerID="ab577f6246421649fbb9e8f5ce62f47ff0856948fc9ce8bac27eca8a24def6d3" Sep 11 14:41:01.606451 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:01.606409 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-lwrt2_openshift-console-operator(57d4ba26-384d-42f8-86a1-335e27cdf762)\"" pod="openshift-console-operator/console-operator-696884fb9c-lwrt2" podUID="57d4ba26-384d-42f8-86a1-335e27cdf762" Sep 11 14:41:01.620486 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:01.620462 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-th99j\" (UID: \"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b\") " pod="openshift-insights/insights-runtime-extractor-th99j" Sep 11 14:41:01.620591 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:01.620516 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-th99j\" (UID: \"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b\") " pod="openshift-insights/insights-runtime-extractor-th99j" Sep 11 14:41:01.620657 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:01.620587 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 14:41:01.620721 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:01.620592 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:41:01.620777 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:01.620669 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-kube-rbac-proxy-cm podName:0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b nodeName:}" failed. No retries permitted until 2026-09-11 14:41:02.620648438 +0000 UTC m=+53.864634088 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-th99j" (UID: "0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b") : configmap "kube-rbac-proxy" not found Sep 11 14:41:01.620777 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:01.620757 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-insights-runtime-extractor-tls podName:0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b nodeName:}" failed. No retries permitted until 2026-09-11 14:41:02.620731708 +0000 UTC m=+53.864717344 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-th99j" (UID: "0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b") : secret "insights-runtime-extractor-tls" not found Sep 11 14:41:02.066822 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:02.066788 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-nzrr9"] Sep 11 14:41:02.068537 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:02.068521 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-nzrr9" Sep 11 14:41:02.070787 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:02.070767 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 11 14:41:02.071675 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:02.071660 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 11 14:41:02.071755 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:02.071697 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-f2gcb\"" Sep 11 14:41:02.079362 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:02.079342 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-nzrr9"] Sep 11 14:41:02.225964 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:02.225930 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxshj\" (UniqueName: \"kubernetes.io/projected/8da9b956-83ea-4049-80e3-e2682aec2402-kube-api-access-bxshj\") pod \"migrator-8b4d697c5-nzrr9\" (UID: \"8da9b956-83ea-4049-80e3-e2682aec2402\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-nzrr9" Sep 11 14:41:02.326515 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:02.326434 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxshj\" (UniqueName: \"kubernetes.io/projected/8da9b956-83ea-4049-80e3-e2682aec2402-kube-api-access-bxshj\") pod \"migrator-8b4d697c5-nzrr9\" (UID: \"8da9b956-83ea-4049-80e3-e2682aec2402\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-nzrr9" Sep 11 14:41:02.336438 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:02.336412 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxshj\" (UniqueName: \"kubernetes.io/projected/8da9b956-83ea-4049-80e3-e2682aec2402-kube-api-access-bxshj\") pod \"migrator-8b4d697c5-nzrr9\" (UID: \"8da9b956-83ea-4049-80e3-e2682aec2402\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-nzrr9" Sep 11 14:41:02.376508 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:02.376474 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-nzrr9" Sep 11 14:41:02.628824 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:02.628799 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-th99j\" (UID: \"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b\") " pod="openshift-insights/insights-runtime-extractor-th99j" Sep 11 14:41:02.629238 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:02.628880 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-th99j\" (UID: \"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b\") " pod="openshift-insights/insights-runtime-extractor-th99j" Sep 11 14:41:02.629238 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:02.628914 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 14:41:02.629238 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:02.628974 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-kube-rbac-proxy-cm podName:0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b nodeName:}" failed. No retries permitted until 2026-09-11 14:41:04.628958668 +0000 UTC m=+55.872944304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-th99j" (UID: "0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b") : configmap "kube-rbac-proxy" not found Sep 11 14:41:02.629238 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:02.629061 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:41:02.629238 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:02.629125 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-insights-runtime-extractor-tls podName:0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b nodeName:}" failed. No retries permitted until 2026-09-11 14:41:04.629109712 +0000 UTC m=+55.873095349 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-th99j" (UID: "0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b") : secret "insights-runtime-extractor-tls" not found Sep 11 14:41:02.647125 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:41:02.647094 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da9b956_83ea_4049_80e3_e2682aec2402.slice/crio-67d0cf1be13db27eb8c1ab642f962db168e918b9282468cf5178ab9f095724e5 WatchSource:0}: Error finding container 67d0cf1be13db27eb8c1ab642f962db168e918b9282468cf5178ab9f095724e5: Status 404 returned error can't find the container with id 67d0cf1be13db27eb8c1ab642f962db168e918b9282468cf5178ab9f095724e5 Sep 11 14:41:02.660346 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:02.660325 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-nzrr9"] Sep 11 14:41:02.804400 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:02.804374 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ppnhx_fa59047c-1d81-44d0-b3a3-9e56bf53511c/dns-node-resolver/0.log" Sep 11 14:41:03.604335 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:03.604302 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4vlml_8dc4e60b-0baa-45d8-93b2-63436c43e3be/node-ca/0.log" Sep 11 14:41:03.612743 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:03.612700 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-nzrr9" event={"ID":"8da9b956-83ea-4049-80e3-e2682aec2402","Type":"ContainerStarted","Data":"67d0cf1be13db27eb8c1ab642f962db168e918b9282468cf5178ab9f095724e5"} Sep 11 14:41:04.342772 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:04.342695 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-registry-tls\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:41:04.343200 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:04.342818 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 14:41:04.343200 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:04.342843 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56f4498d5f-4dwl8: secret "image-registry-tls" not found Sep 11 14:41:04.343200 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:04.342900 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-registry-tls podName:c6f478f4-6ae4-4ed8-8881-b1945228d966 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:12.342884718 +0000 UTC m=+63.586870382 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-registry-tls") pod "image-registry-56f4498d5f-4dwl8" (UID: "c6f478f4-6ae4-4ed8-8881-b1945228d966") : secret "image-registry-tls" not found Sep 11 14:41:04.603863 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:04.603743 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ht6zm_062d13b8-8971-4c91-b348-8b609ed42b56/serve-healthcheck-canary/0.log" Sep 11 14:41:04.616314 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:04.616281 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-nzrr9" event={"ID":"8da9b956-83ea-4049-80e3-e2682aec2402","Type":"ContainerStarted","Data":"c0fd1380d93fc4e4b32828b3af71e9316cede44d7364b1d7787172ad99a25189"} Sep 11 14:41:04.616451 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:04.616331 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-nzrr9" event={"ID":"8da9b956-83ea-4049-80e3-e2682aec2402","Type":"ContainerStarted","Data":"d38d8a0d5d347d59beffa3d39b436c2c4888d67188bcb3b705caa0920ca05875"} Sep 11 14:41:04.635351 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:04.635300 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-nzrr9" podStartSLOduration=1.479316138 podStartE2EDuration="2.635284122s" totalImagesPullingTime="1.155967984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:04.634234899 +0000 UTC m=+55.878220557" watchObservedRunningTime="2026-09-11 14:41:04.635284122 +0000 UTC m=+55.879269781" Sep 11 14:41:04.644807 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:04.644783 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-th99j\" (UID: \"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b\") " pod="openshift-insights/insights-runtime-extractor-th99j" Sep 11 14:41:04.644903 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:04.644841 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-th99j\" (UID: \"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b\") " pod="openshift-insights/insights-runtime-extractor-th99j" Sep 11 14:41:04.644937 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:04.644905 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 14:41:04.644974 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:04.644941 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:41:04.645003 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:04.644979 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-kube-rbac-proxy-cm podName:0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b nodeName:}" failed. No retries permitted until 2026-09-11 14:41:08.644959636 +0000 UTC m=+59.888945286 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-th99j" (UID: "0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b") : configmap "kube-rbac-proxy" not found Sep 11 14:41:04.645003 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:04.644993 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-insights-runtime-extractor-tls podName:0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b nodeName:}" failed. No retries permitted until 2026-09-11 14:41:08.644987332 +0000 UTC m=+59.888972968 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-th99j" (UID: "0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b") : secret "insights-runtime-extractor-tls" not found Sep 11 14:41:06.301557 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:06.301516 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-lwrt2" Sep 11 14:41:06.301950 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:06.301575 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-lwrt2" Sep 11 14:41:06.301950 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:06.301926 2583 scope.go:117] "RemoveContainer" containerID="ab577f6246421649fbb9e8f5ce62f47ff0856948fc9ce8bac27eca8a24def6d3" Sep 11 14:41:06.302085 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:06.302065 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-lwrt2_openshift-console-operator(57d4ba26-384d-42f8-86a1-335e27cdf762)\"" pod="openshift-console-operator/console-operator-696884fb9c-lwrt2" podUID="57d4ba26-384d-42f8-86a1-335e27cdf762" Sep 11 14:41:08.539595 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:08.539560 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vlg7j" Sep 11 14:41:08.675618 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:08.675585 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-th99j\" (UID: \"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b\") " pod="openshift-insights/insights-runtime-extractor-th99j" Sep 11 14:41:08.675814 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:08.675652 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-th99j\" (UID: \"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b\") " pod="openshift-insights/insights-runtime-extractor-th99j" Sep 11 14:41:08.675814 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:08.675755 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 14:41:08.675814 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:08.675759 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:41:08.675814 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:08.675812 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-kube-rbac-proxy-cm podName:0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b nodeName:}" failed. No retries permitted until 2026-09-11 14:41:16.675797446 +0000 UTC m=+67.919783086 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-th99j" (UID: "0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b") : configmap "kube-rbac-proxy" not found Sep 11 14:41:08.675981 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:08.675826 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-insights-runtime-extractor-tls podName:0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b nodeName:}" failed. No retries permitted until 2026-09-11 14:41:16.675820396 +0000 UTC m=+67.919806034 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-th99j" (UID: "0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b") : secret "insights-runtime-extractor-tls" not found Sep 11 14:41:12.404014 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:12.403972 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-registry-tls\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:41:12.406230 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:12.406208 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-registry-tls\") pod \"image-registry-56f4498d5f-4dwl8\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:41:12.410414 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:12.410392 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6xrxr\"" Sep 11 14:41:12.418402 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:12.418386 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:41:12.692602 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:12.692512 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56f4498d5f-4dwl8"] Sep 11 14:41:12.696945 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:41:12.696912 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f478f4_6ae4_4ed8_8881_b1945228d966.slice/crio-662ab23cff42a14b81fc3ec1de600a3e6d1382ebd98941c37e554d8e73cf7ffb WatchSource:0}: Error finding container 662ab23cff42a14b81fc3ec1de600a3e6d1382ebd98941c37e554d8e73cf7ffb: Status 404 returned error can't find the container with id 662ab23cff42a14b81fc3ec1de600a3e6d1382ebd98941c37e554d8e73cf7ffb Sep 11 14:41:13.638594 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:13.638553 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" event={"ID":"c6f478f4-6ae4-4ed8-8881-b1945228d966","Type":"ContainerStarted","Data":"9b702ad63e4376a980e5067920e10d6f100787536b63886b2a6997b38f60cb76"} Sep 11 14:41:13.638996 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:13.638609 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" event={"ID":"c6f478f4-6ae4-4ed8-8881-b1945228d966","Type":"ContainerStarted","Data":"662ab23cff42a14b81fc3ec1de600a3e6d1382ebd98941c37e554d8e73cf7ffb"} Sep 11 14:41:13.638996 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:13.638655 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:41:13.660039 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:13.659992 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" podStartSLOduration=17.65997775 podStartE2EDuration="17.65997775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:13.658847777 +0000 UTC m=+64.902833436" watchObservedRunningTime="2026-09-11 14:41:13.65997775 +0000 UTC m=+64.903963407" Sep 11 14:41:14.520677 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:14.520634 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70ec3434-0593-4acd-bd81-2580090a5020-metrics-tls\") pod \"dns-default-7mrb4\" (UID: \"70ec3434-0593-4acd-bd81-2580090a5020\") " pod="openshift-dns/dns-default-7mrb4" Sep 11 14:41:14.523042 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:14.523006 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/70ec3434-0593-4acd-bd81-2580090a5020-metrics-tls\") pod \"dns-default-7mrb4\" (UID: \"70ec3434-0593-4acd-bd81-2580090a5020\") " pod="openshift-dns/dns-default-7mrb4" Sep 11 14:41:14.755815 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:14.755784 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vx9kl\"" Sep 11 14:41:14.763788 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:14.763769 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7mrb4" Sep 11 14:41:15.024591 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:15.024512 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-original-pull-secret\") pod \"global-pull-secret-syncer-mlrnk\" (UID: \"a1b523f4-d191-49bb-8e0e-4999e6fa1a92\") " pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:41:15.024591 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:15.024563 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8g5r\" (UniqueName: \"kubernetes.io/projected/ecdbcfa3-6309-46be-a9c9-ea8eb658de34-kube-api-access-p8g5r\") pod \"network-check-target-5hp9w\" (UID: \"ecdbcfa3-6309-46be-a9c9-ea8eb658de34\") " pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:41:15.024851 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:15.024617 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73182ce3-cb4b-4fcb-992e-85d116cf7e15-metrics-certs\") pod \"network-metrics-daemon-brx7q\" (UID: \"73182ce3-cb4b-4fcb-992e-85d116cf7e15\") " pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:41:15.026949 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:15.026927 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 14:41:15.027052 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:15.026994 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 14:41:15.027589 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:15.027573 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 14:41:15.033203 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:15.033173 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7mrb4"] Sep 11 14:41:15.036900 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:41:15.036863 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ec3434_0593_4acd_bd81_2580090a5020.slice/crio-a2b41506acfafa4602dda3ca7e202e1d697e5cdb4d64cf8718e6cb33632e6d23 WatchSource:0}: Error finding container a2b41506acfafa4602dda3ca7e202e1d697e5cdb4d64cf8718e6cb33632e6d23: Status 404 returned error can't find the container with id a2b41506acfafa4602dda3ca7e202e1d697e5cdb4d64cf8718e6cb33632e6d23 Sep 11 14:41:15.037398 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:15.037368 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a1b523f4-d191-49bb-8e0e-4999e6fa1a92-original-pull-secret\") pod \"global-pull-secret-syncer-mlrnk\" (UID: \"a1b523f4-d191-49bb-8e0e-4999e6fa1a92\") " pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:41:15.037531 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:15.037511 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73182ce3-cb4b-4fcb-992e-85d116cf7e15-metrics-certs\") pod \"network-metrics-daemon-brx7q\" (UID: \"73182ce3-cb4b-4fcb-992e-85d116cf7e15\") " pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:41:15.037809 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:15.037789 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 14:41:15.047465 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:15.047446 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8g5r\" (UniqueName: \"kubernetes.io/projected/ecdbcfa3-6309-46be-a9c9-ea8eb658de34-kube-api-access-p8g5r\") pod \"network-check-target-5hp9w\" (UID: \"ecdbcfa3-6309-46be-a9c9-ea8eb658de34\") " pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:41:15.137755 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:15.137728 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jjd7d\"" Sep 11 14:41:15.145913 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:15.145895 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brx7q" Sep 11 14:41:15.239964 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:15.239932 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xdw2d\"" Sep 11 14:41:15.241868 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:15.241852 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlrnk" Sep 11 14:41:15.247637 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:15.247616 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:41:15.418909 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:15.418885 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-brx7q"] Sep 11 14:41:15.419583 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:41:15.419557 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73182ce3_cb4b_4fcb_992e_85d116cf7e15.slice/crio-76497589afff2584740c3aa71097fe588cde82de7d5be18c853bcfcb634bccd3 WatchSource:0}: Error finding container 76497589afff2584740c3aa71097fe588cde82de7d5be18c853bcfcb634bccd3: Status 404 returned error can't find the container with id 76497589afff2584740c3aa71097fe588cde82de7d5be18c853bcfcb634bccd3 Sep 11 14:41:15.520453 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:15.520417 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mlrnk"] Sep 11 14:41:15.523644 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:41:15.523612 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b523f4_d191_49bb_8e0e_4999e6fa1a92.slice/crio-88c398550aed6158b0787305278cdd50dde8942983c123a2f69c69c6fd09926e WatchSource:0}: Error finding container 88c398550aed6158b0787305278cdd50dde8942983c123a2f69c69c6fd09926e: Status 404 returned error can't find the container with id 88c398550aed6158b0787305278cdd50dde8942983c123a2f69c69c6fd09926e Sep 11 14:41:15.529734 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:15.529671 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5hp9w"] Sep 11 14:41:15.533408 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:41:15.533381 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecdbcfa3_6309_46be_a9c9_ea8eb658de34.slice/crio-5dd38d75fd17ab83b95e86409b530d133c3ea8240f71434cde46bd6f9913cb7e WatchSource:0}: Error finding container 5dd38d75fd17ab83b95e86409b530d133c3ea8240f71434cde46bd6f9913cb7e: Status 404 returned error can't find the container with id 5dd38d75fd17ab83b95e86409b530d133c3ea8240f71434cde46bd6f9913cb7e Sep 11 14:41:15.644876 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:15.644835 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5hp9w" event={"ID":"ecdbcfa3-6309-46be-a9c9-ea8eb658de34","Type":"ContainerStarted","Data":"5dd38d75fd17ab83b95e86409b530d133c3ea8240f71434cde46bd6f9913cb7e"} Sep 11 14:41:15.646164 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:15.646133 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mlrnk" event={"ID":"a1b523f4-d191-49bb-8e0e-4999e6fa1a92","Type":"ContainerStarted","Data":"88c398550aed6158b0787305278cdd50dde8942983c123a2f69c69c6fd09926e"} Sep 11 14:41:15.647496 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:15.647470 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-brx7q" event={"ID":"73182ce3-cb4b-4fcb-992e-85d116cf7e15","Type":"ContainerStarted","Data":"76497589afff2584740c3aa71097fe588cde82de7d5be18c853bcfcb634bccd3"} Sep 11 14:41:15.648689 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:15.648668 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7mrb4" event={"ID":"70ec3434-0593-4acd-bd81-2580090a5020","Type":"ContainerStarted","Data":"a2b41506acfafa4602dda3ca7e202e1d697e5cdb4d64cf8718e6cb33632e6d23"} Sep 11 14:41:16.739993 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:16.739948 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-th99j\" (UID: \"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b\") " pod="openshift-insights/insights-runtime-extractor-th99j" Sep 11 14:41:16.740421 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:16.740042 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-th99j\" (UID: \"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b\") " pod="openshift-insights/insights-runtime-extractor-th99j" Sep 11 14:41:16.741001 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:16.740975 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-th99j\" (UID: \"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b\") " pod="openshift-insights/insights-runtime-extractor-th99j" Sep 11 14:41:16.743420 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:16.743395 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-th99j\" (UID: \"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b\") " pod="openshift-insights/insights-runtime-extractor-th99j" Sep 11 14:41:16.809802 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:16.809763 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6tvzn\"" Sep 11 14:41:16.818241 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:16.818209 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-th99j" Sep 11 14:41:17.315313 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:17.315190 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-th99j"] Sep 11 14:41:17.318575 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:41:17.318543 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fdaf3a0_5d6f_43e8_8746_6fd8698a7d9b.slice/crio-b9483b78f31ff423e54d665474deb86750fdecabd4a60cef3c81d3bc5cddf494 WatchSource:0}: Error finding container b9483b78f31ff423e54d665474deb86750fdecabd4a60cef3c81d3bc5cddf494: Status 404 returned error can't find the container with id b9483b78f31ff423e54d665474deb86750fdecabd4a60cef3c81d3bc5cddf494 Sep 11 14:41:17.660021 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:17.659581 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-brx7q" event={"ID":"73182ce3-cb4b-4fcb-992e-85d116cf7e15","Type":"ContainerStarted","Data":"62394c35d4887430f35ec9cff08f15a5e21453c77ce81c5602f4dd2b733bb109"} Sep 11 14:41:17.660021 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:17.659652 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-brx7q" event={"ID":"73182ce3-cb4b-4fcb-992e-85d116cf7e15","Type":"ContainerStarted","Data":"bcef166e887f7607d6f3af287181b6160e0180c03f8a847472b493d4afc4cb6d"} Sep 11 14:41:17.661414 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:17.661382 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-th99j" event={"ID":"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b","Type":"ContainerStarted","Data":"a96fa49a934210e02e6addfd4f764b60de53d2c36f6603f0dbb215b5a2adde3e"} Sep 11 14:41:17.661539 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:17.661453 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-th99j" event={"ID":"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b","Type":"ContainerStarted","Data":"b9483b78f31ff423e54d665474deb86750fdecabd4a60cef3c81d3bc5cddf494"} Sep 11 14:41:17.663565 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:17.663533 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7mrb4" event={"ID":"70ec3434-0593-4acd-bd81-2580090a5020","Type":"ContainerStarted","Data":"570aaccd5876b2922db1dd328b6885716c67d467dbbe658552e65237f038b8ef"} Sep 11 14:41:17.663670 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:17.663590 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7mrb4" event={"ID":"70ec3434-0593-4acd-bd81-2580090a5020","Type":"ContainerStarted","Data":"0cb2e3180d2f7aff06ff01d529de235d5d961af3d84e71b43b74ec702df76e45"} Sep 11 14:41:17.663817 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:17.663791 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7mrb4" Sep 11 14:41:17.677566 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:17.677527 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-brx7q" podStartSLOduration=67.072645997 podStartE2EDuration="1m8.67751453s" totalImagesPullingTime="1.604868533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:17.676131121 +0000 UTC m=+68.920116779" watchObservedRunningTime="2026-09-11 14:41:17.67751453 +0000 UTC m=+68.921500187" Sep 11 14:41:17.703481 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:17.703426 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7mrb4" podStartSLOduration=33.719066994 podStartE2EDuration="35.703407327s" totalImagesPullingTime="1.984340333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:17.701477903 +0000 UTC m=+68.945463563" watchObservedRunningTime="2026-09-11 14:41:17.703407327 +0000 UTC m=+68.947392988" Sep 11 14:41:18.324617 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:18.324588 2583 scope.go:117] "RemoveContainer" containerID="ab577f6246421649fbb9e8f5ce62f47ff0856948fc9ce8bac27eca8a24def6d3" Sep 11 14:41:21.681071 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:21.681027 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-th99j" event={"ID":"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b","Type":"ContainerStarted","Data":"3efd3cf592bbcee1b656b6c1a640e9f160a44bbab4ba2f6e1675c04fa2d40f65"} Sep 11 14:41:21.683361 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:21.683332 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lwrt2_57d4ba26-384d-42f8-86a1-335e27cdf762/console-operator/1.log" Sep 11 14:41:21.683499 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:21.683460 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-lwrt2" event={"ID":"57d4ba26-384d-42f8-86a1-335e27cdf762","Type":"ContainerStarted","Data":"c2368e1e50c385032cbcffb105f1570435f2b8f00ffdfcedb041ed2e4f2f48c4"} Sep 11 14:41:21.683835 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:21.683812 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-lwrt2" Sep 11 14:41:21.685752 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:21.685728 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5hp9w" event={"ID":"ecdbcfa3-6309-46be-a9c9-ea8eb658de34","Type":"ContainerStarted","Data":"b832365f6807d1296b86376679158369496a43b7049c2c0ed134dffdb9d2272a"} Sep 11 14:41:21.685866 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:21.685829 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:41:21.687914 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:21.687890 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mlrnk" event={"ID":"a1b523f4-d191-49bb-8e0e-4999e6fa1a92","Type":"ContainerStarted","Data":"a36a2e67fc411217ffa641b36da7bfb07b28bfffd44229fa21213a5e68c06fcc"} Sep 11 14:41:21.697473 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:21.697450 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-lwrt2" Sep 11 14:41:21.703253 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:21.703210 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-lwrt2" podStartSLOduration=24.167928513 podStartE2EDuration="26.703192578s" totalImagesPullingTime="2.535264065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:21.702110302 +0000 UTC m=+72.946095960" watchObservedRunningTime="2026-09-11 14:41:21.703192578 +0000 UTC m=+72.947178238" Sep 11 14:41:21.720334 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:21.720275 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mlrnk" podStartSLOduration=67.418400405 podStartE2EDuration="1m12.720260673s" totalImagesPullingTime="5.301860268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:21.719807525 +0000 UTC m=+72.963793181" watchObservedRunningTime="2026-09-11 14:41:21.720260673 +0000 UTC m=+72.964246363" Sep 11 14:41:21.741917 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:21.741859 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5hp9w" podStartSLOduration=67.454513482 podStartE2EDuration="1m12.741844624s" totalImagesPullingTime="5.287331142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:21.737383063 +0000 UTC m=+72.981368736" watchObservedRunningTime="2026-09-11 14:41:21.741844624 +0000 UTC m=+72.985830283" Sep 11 14:41:22.693536 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:22.693493 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-th99j" event={"ID":"0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b","Type":"ContainerStarted","Data":"fb2f8ae10024d5a1ed2d6915b3464408b6b4a279f7728e313ad606dc2bb24a90"} Sep 11 14:41:22.713561 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:22.713504 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-th99j" podStartSLOduration=20.788516525 podStartE2EDuration="22.713488812s" totalImagesPullingTime="1.924972287s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:41:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:22.712054178 +0000 UTC m=+73.956039835" watchObservedRunningTime="2026-09-11 14:41:22.713488812 +0000 UTC m=+73.957474471" Sep 11 14:41:27.636770 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.636732 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm"] Sep 11 14:41:27.638713 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.638682 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm" Sep 11 14:41:27.641029 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.641002 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 11 14:41:27.641378 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.641359 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 11 14:41:27.641587 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.641566 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 11 14:41:27.641716 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.641688 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 11 14:41:27.641921 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.641908 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 11 14:41:27.641980 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.641947 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 11 14:41:27.642247 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.642231 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 11 14:41:27.655898 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.655864 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm"] Sep 11 14:41:27.669760 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.669742 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7mrb4" Sep 11 14:41:27.688846 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.688807 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56f4498d5f-4dwl8"] Sep 11 14:41:27.695146 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.695119 2583 patch_prober.go:28] interesting pod/image-registry-56f4498d5f-4dwl8 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 11 14:41:27.695231 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.695168 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" podUID="c6f478f4-6ae4-4ed8-8881-b1945228d966" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:41:27.717989 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.717961 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/03baf50a-07a1-43fc-9608-ebec034ba067-additional-service-ca\") pod \"cluster-proxy-proxy-agent-55b994c6d4-v6klm\" (UID: \"03baf50a-07a1-43fc-9608-ebec034ba067\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm" Sep 11 14:41:27.718111 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.718009 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/03baf50a-07a1-43fc-9608-ebec034ba067-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-55b994c6d4-v6klm\" (UID: \"03baf50a-07a1-43fc-9608-ebec034ba067\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm" Sep 11 14:41:27.718111 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.718049 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/03baf50a-07a1-43fc-9608-ebec034ba067-ca\") pod \"cluster-proxy-proxy-agent-55b994c6d4-v6klm\" (UID: \"03baf50a-07a1-43fc-9608-ebec034ba067\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm" Sep 11 14:41:27.718111 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.718064 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptkzc\" (UniqueName: \"kubernetes.io/projected/03baf50a-07a1-43fc-9608-ebec034ba067-kube-api-access-ptkzc\") pod \"cluster-proxy-proxy-agent-55b994c6d4-v6klm\" (UID: \"03baf50a-07a1-43fc-9608-ebec034ba067\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm" Sep 11 14:41:27.718284 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.718185 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/03baf50a-07a1-43fc-9608-ebec034ba067-hub\") pod \"cluster-proxy-proxy-agent-55b994c6d4-v6klm\" (UID: \"03baf50a-07a1-43fc-9608-ebec034ba067\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm" Sep 11 14:41:27.718337 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.718294 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/03baf50a-07a1-43fc-9608-ebec034ba067-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-55b994c6d4-v6klm\" (UID: \"03baf50a-07a1-43fc-9608-ebec034ba067\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm" Sep 11 14:41:27.818965 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.818931 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/03baf50a-07a1-43fc-9608-ebec034ba067-ca\") pod \"cluster-proxy-proxy-agent-55b994c6d4-v6klm\" (UID: \"03baf50a-07a1-43fc-9608-ebec034ba067\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm" Sep 11 14:41:27.818965 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.818967 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptkzc\" (UniqueName: \"kubernetes.io/projected/03baf50a-07a1-43fc-9608-ebec034ba067-kube-api-access-ptkzc\") pod \"cluster-proxy-proxy-agent-55b994c6d4-v6klm\" (UID: \"03baf50a-07a1-43fc-9608-ebec034ba067\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm" Sep 11 14:41:27.819193 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.818996 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/03baf50a-07a1-43fc-9608-ebec034ba067-hub\") pod \"cluster-proxy-proxy-agent-55b994c6d4-v6klm\" (UID: \"03baf50a-07a1-43fc-9608-ebec034ba067\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm" Sep 11 14:41:27.819193 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.819029 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/03baf50a-07a1-43fc-9608-ebec034ba067-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-55b994c6d4-v6klm\" (UID: \"03baf50a-07a1-43fc-9608-ebec034ba067\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm" Sep 11 14:41:27.819193 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.819064 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/03baf50a-07a1-43fc-9608-ebec034ba067-additional-service-ca\") pod \"cluster-proxy-proxy-agent-55b994c6d4-v6klm\" (UID: \"03baf50a-07a1-43fc-9608-ebec034ba067\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm" Sep 11 14:41:27.819193 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.819089 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/03baf50a-07a1-43fc-9608-ebec034ba067-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-55b994c6d4-v6klm\" (UID: \"03baf50a-07a1-43fc-9608-ebec034ba067\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm" Sep 11 14:41:27.819826 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.819796 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/03baf50a-07a1-43fc-9608-ebec034ba067-additional-service-ca\") pod \"cluster-proxy-proxy-agent-55b994c6d4-v6klm\" (UID: \"03baf50a-07a1-43fc-9608-ebec034ba067\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm" Sep 11 14:41:27.821602 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.821580 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/03baf50a-07a1-43fc-9608-ebec034ba067-hub\") pod \"cluster-proxy-proxy-agent-55b994c6d4-v6klm\" (UID: \"03baf50a-07a1-43fc-9608-ebec034ba067\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm" Sep 11 14:41:27.821719 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.821609 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/03baf50a-07a1-43fc-9608-ebec034ba067-ca\") pod \"cluster-proxy-proxy-agent-55b994c6d4-v6klm\" (UID: \"03baf50a-07a1-43fc-9608-ebec034ba067\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm" Sep 11 14:41:27.821719 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.821633 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/03baf50a-07a1-43fc-9608-ebec034ba067-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-55b994c6d4-v6klm\" (UID: \"03baf50a-07a1-43fc-9608-ebec034ba067\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm" Sep 11 14:41:27.821802 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.821734 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/03baf50a-07a1-43fc-9608-ebec034ba067-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-55b994c6d4-v6klm\" (UID: \"03baf50a-07a1-43fc-9608-ebec034ba067\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm" Sep 11 14:41:27.838089 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.838064 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptkzc\" (UniqueName: \"kubernetes.io/projected/03baf50a-07a1-43fc-9608-ebec034ba067-kube-api-access-ptkzc\") pod \"cluster-proxy-proxy-agent-55b994c6d4-v6klm\" (UID: \"03baf50a-07a1-43fc-9608-ebec034ba067\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm" Sep 11 14:41:27.958914 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:27.958835 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm" Sep 11 14:41:28.260376 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:28.260302 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm"] Sep 11 14:41:28.263452 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:41:28.263422 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03baf50a_07a1_43fc_9608_ebec034ba067.slice/crio-ea0ad03f01cf2a454ca97b201aeffbdfbb1ab76b0d3f1bbdccfe8135c25ad680 WatchSource:0}: Error finding container ea0ad03f01cf2a454ca97b201aeffbdfbb1ab76b0d3f1bbdccfe8135c25ad680: Status 404 returned error can't find the container with id ea0ad03f01cf2a454ca97b201aeffbdfbb1ab76b0d3f1bbdccfe8135c25ad680 Sep 11 14:41:28.714889 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:28.714856 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm" event={"ID":"03baf50a-07a1-43fc-9608-ebec034ba067","Type":"ContainerStarted","Data":"ea0ad03f01cf2a454ca97b201aeffbdfbb1ab76b0d3f1bbdccfe8135c25ad680"} Sep 11 14:41:32.730336 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:32.730296 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm" event={"ID":"03baf50a-07a1-43fc-9608-ebec034ba067","Type":"ContainerStarted","Data":"07326c9f0476b6dfb17e8c508fb35667d504c414705368649d76bdfc2ed0d8f3"} Sep 11 14:41:32.730788 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:32.730343 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm" event={"ID":"03baf50a-07a1-43fc-9608-ebec034ba067","Type":"ContainerStarted","Data":"0d5d8eb09231b6e85fa8a8ff9be1f74231630309033b2853864d8f0dfa730621"} Sep 11 14:41:32.730788 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:32.730359 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm" event={"ID":"03baf50a-07a1-43fc-9608-ebec034ba067","Type":"ContainerStarted","Data":"2391f1f9a37690144b653e36717626e88f30c24665802969272c21be93689572"} Sep 11 14:41:32.782186 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:32.782138 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55b994c6d4-v6klm" podStartSLOduration=2.360418016 podStartE2EDuration="5.782125124s" totalImagesPullingTime="3.421707108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:41:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:32.781950804 +0000 UTC m=+84.025936463" watchObservedRunningTime="2026-09-11 14:41:32.782125124 +0000 UTC m=+84.026110781" Sep 11 14:41:37.693441 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:37.693413 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:41:49.430178 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.430146 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fp8dm"] Sep 11 14:41:49.461352 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.461325 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:49.465647 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.465622 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9pcp2\"" Sep 11 14:41:49.466101 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.466086 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 11 14:41:49.466157 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.466136 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 11 14:41:49.466781 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.466764 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 11 14:41:49.466781 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.466773 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 11 14:41:49.466903 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.466876 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 11 14:41:49.466961 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.466946 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 11 14:41:49.577214 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.577182 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8391c39-3072-4cba-9812-5dc45d82589d-sys\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:49.577396 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.577228 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8391c39-3072-4cba-9812-5dc45d82589d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:49.577396 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.577300 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c8391c39-3072-4cba-9812-5dc45d82589d-node-exporter-accelerators-collector-config\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:49.577396 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.577365 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c8391c39-3072-4cba-9812-5dc45d82589d-node-exporter-wtmp\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:49.577521 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.577400 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vrmr\" (UniqueName: \"kubernetes.io/projected/c8391c39-3072-4cba-9812-5dc45d82589d-kube-api-access-7vrmr\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:49.577521 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.577428 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8391c39-3072-4cba-9812-5dc45d82589d-metrics-client-ca\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:49.577521 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.577454 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c8391c39-3072-4cba-9812-5dc45d82589d-node-exporter-tls\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:49.577521 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.577480 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c8391c39-3072-4cba-9812-5dc45d82589d-root\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:49.577521 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.577496 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c8391c39-3072-4cba-9812-5dc45d82589d-node-exporter-textfile\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:49.678605 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.678566 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c8391c39-3072-4cba-9812-5dc45d82589d-node-exporter-accelerators-collector-config\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:49.678825 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.678612 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c8391c39-3072-4cba-9812-5dc45d82589d-node-exporter-wtmp\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:49.678825 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.678767 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vrmr\" (UniqueName: \"kubernetes.io/projected/c8391c39-3072-4cba-9812-5dc45d82589d-kube-api-access-7vrmr\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:49.678825 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.678802 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8391c39-3072-4cba-9812-5dc45d82589d-metrics-client-ca\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:49.678825 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.678813 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c8391c39-3072-4cba-9812-5dc45d82589d-node-exporter-wtmp\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:49.679037 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.678819 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c8391c39-3072-4cba-9812-5dc45d82589d-node-exporter-tls\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:49.679037 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.678886 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c8391c39-3072-4cba-9812-5dc45d82589d-root\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:49.679037 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.678904 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c8391c39-3072-4cba-9812-5dc45d82589d-node-exporter-textfile\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:49.679037 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.678970 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c8391c39-3072-4cba-9812-5dc45d82589d-root\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:49.679037 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.678974 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8391c39-3072-4cba-9812-5dc45d82589d-sys\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:49.679037 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.679004 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8391c39-3072-4cba-9812-5dc45d82589d-sys\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:49.679037 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.679034 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8391c39-3072-4cba-9812-5dc45d82589d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:49.679375 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:49.678996 2583 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 11 14:41:49.679375 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:49.679179 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8391c39-3072-4cba-9812-5dc45d82589d-node-exporter-tls podName:c8391c39-3072-4cba-9812-5dc45d82589d nodeName:}" failed. No retries permitted until 2026-09-11 14:41:50.179143171 +0000 UTC m=+101.423128828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c8391c39-3072-4cba-9812-5dc45d82589d-node-exporter-tls") pod "node-exporter-fp8dm" (UID: "c8391c39-3072-4cba-9812-5dc45d82589d") : secret "node-exporter-tls" not found Sep 11 14:41:49.679375 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.679285 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c8391c39-3072-4cba-9812-5dc45d82589d-node-exporter-textfile\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:49.679375 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.679304 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c8391c39-3072-4cba-9812-5dc45d82589d-node-exporter-accelerators-collector-config\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:49.679375 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.679339 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8391c39-3072-4cba-9812-5dc45d82589d-metrics-client-ca\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:49.681262 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.681224 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8391c39-3072-4cba-9812-5dc45d82589d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:49.695940 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:49.695904 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vrmr\" (UniqueName: \"kubernetes.io/projected/c8391c39-3072-4cba-9812-5dc45d82589d-kube-api-access-7vrmr\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:50.183044 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:50.183005 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c8391c39-3072-4cba-9812-5dc45d82589d-node-exporter-tls\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:50.185252 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:50.185234 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c8391c39-3072-4cba-9812-5dc45d82589d-node-exporter-tls\") pod \"node-exporter-fp8dm\" (UID: \"c8391c39-3072-4cba-9812-5dc45d82589d\") " pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:50.370465 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:50.370435 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fp8dm" Sep 11 14:41:50.413084 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:41:50.413054 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8391c39_3072_4cba_9812_5dc45d82589d.slice/crio-30c7cc43e7ee6a2a16a34229b2203740afe0dcecf94ef41e2adcdafdebff6645 WatchSource:0}: Error finding container 30c7cc43e7ee6a2a16a34229b2203740afe0dcecf94ef41e2adcdafdebff6645: Status 404 returned error can't find the container with id 30c7cc43e7ee6a2a16a34229b2203740afe0dcecf94ef41e2adcdafdebff6645 Sep 11 14:41:50.788676 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:50.788645 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fp8dm" event={"ID":"c8391c39-3072-4cba-9812-5dc45d82589d","Type":"ContainerStarted","Data":"30c7cc43e7ee6a2a16a34229b2203740afe0dcecf94ef41e2adcdafdebff6645"} Sep 11 14:41:52.696935 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:52.696904 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5hp9w" Sep 11 14:41:52.708619 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:52.708586 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" podUID="c6f478f4-6ae4-4ed8-8881-b1945228d966" containerName="registry" containerID="cri-o://9b702ad63e4376a980e5067920e10d6f100787536b63886b2a6997b38f60cb76" gracePeriod=30 Sep 11 14:41:52.799656 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:52.799626 2583 generic.go:356] "Generic (PLEG): container finished" podID="c8391c39-3072-4cba-9812-5dc45d82589d" containerID="9fb8ee605dcdc0b6f367c2bc843184f17102c096f9dd865beb89a5632551c09c" exitCode=0 Sep 11 14:41:52.799805 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:52.799663 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fp8dm" event={"ID":"c8391c39-3072-4cba-9812-5dc45d82589d","Type":"ContainerDied","Data":"9fb8ee605dcdc0b6f367c2bc843184f17102c096f9dd865beb89a5632551c09c"} Sep 11 14:41:52.975735 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:52.975691 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:41:53.103965 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.103934 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6f478f4-6ae4-4ed8-8881-b1945228d966-installation-pull-secrets\") pod \"c6f478f4-6ae4-4ed8-8881-b1945228d966\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " Sep 11 14:41:53.104164 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.103973 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6f478f4-6ae4-4ed8-8881-b1945228d966-registry-certificates\") pod \"c6f478f4-6ae4-4ed8-8881-b1945228d966\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " Sep 11 14:41:53.104164 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.104029 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmg2g\" (UniqueName: \"kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-kube-api-access-jmg2g\") pod \"c6f478f4-6ae4-4ed8-8881-b1945228d966\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " Sep 11 14:41:53.104164 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.104071 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6f478f4-6ae4-4ed8-8881-b1945228d966-ca-trust-extracted\") pod \"c6f478f4-6ae4-4ed8-8881-b1945228d966\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " Sep 11 14:41:53.104164 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.104104 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-bound-sa-token\") pod \"c6f478f4-6ae4-4ed8-8881-b1945228d966\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " Sep 11 14:41:53.104164 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.104129 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6f478f4-6ae4-4ed8-8881-b1945228d966-trusted-ca\") pod \"c6f478f4-6ae4-4ed8-8881-b1945228d966\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " Sep 11 14:41:53.104164 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.104150 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-registry-tls\") pod \"c6f478f4-6ae4-4ed8-8881-b1945228d966\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " Sep 11 14:41:53.104433 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.104183 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6f478f4-6ae4-4ed8-8881-b1945228d966-image-registry-private-configuration\") pod \"c6f478f4-6ae4-4ed8-8881-b1945228d966\" (UID: \"c6f478f4-6ae4-4ed8-8881-b1945228d966\") " Sep 11 14:41:53.104506 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.104477 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f478f4-6ae4-4ed8-8881-b1945228d966-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c6f478f4-6ae4-4ed8-8881-b1945228d966" (UID: "c6f478f4-6ae4-4ed8-8881-b1945228d966"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:41:53.104615 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.104587 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f478f4-6ae4-4ed8-8881-b1945228d966-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c6f478f4-6ae4-4ed8-8881-b1945228d966" (UID: "c6f478f4-6ae4-4ed8-8881-b1945228d966"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:41:53.106551 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.106499 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6f478f4-6ae4-4ed8-8881-b1945228d966-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c6f478f4-6ae4-4ed8-8881-b1945228d966" (UID: "c6f478f4-6ae4-4ed8-8881-b1945228d966"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:41:53.106551 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.106534 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-kube-api-access-jmg2g" (OuterVolumeSpecName: "kube-api-access-jmg2g") pod "c6f478f4-6ae4-4ed8-8881-b1945228d966" (UID: "c6f478f4-6ae4-4ed8-8881-b1945228d966"). InnerVolumeSpecName "kube-api-access-jmg2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:41:53.106771 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.106643 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6f478f4-6ae4-4ed8-8881-b1945228d966-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c6f478f4-6ae4-4ed8-8881-b1945228d966" (UID: "c6f478f4-6ae4-4ed8-8881-b1945228d966"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:41:53.106879 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.106860 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c6f478f4-6ae4-4ed8-8881-b1945228d966" (UID: "c6f478f4-6ae4-4ed8-8881-b1945228d966"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:41:53.106976 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.106952 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c6f478f4-6ae4-4ed8-8881-b1945228d966" (UID: "c6f478f4-6ae4-4ed8-8881-b1945228d966"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:41:53.112778 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.112754 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f478f4-6ae4-4ed8-8881-b1945228d966-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c6f478f4-6ae4-4ed8-8881-b1945228d966" (UID: "c6f478f4-6ae4-4ed8-8881-b1945228d966"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:41:53.205392 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.205336 2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6f478f4-6ae4-4ed8-8881-b1945228d966-ca-trust-extracted\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:41:53.205392 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.205358 2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-bound-sa-token\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:41:53.205392 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.205367 2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6f478f4-6ae4-4ed8-8881-b1945228d966-trusted-ca\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:41:53.205392 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.205375 2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-registry-tls\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:41:53.205392 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.205383 2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6f478f4-6ae4-4ed8-8881-b1945228d966-image-registry-private-configuration\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:41:53.205392 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.205391 2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6f478f4-6ae4-4ed8-8881-b1945228d966-installation-pull-secrets\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:41:53.205688 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.205401 2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6f478f4-6ae4-4ed8-8881-b1945228d966-registry-certificates\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:41:53.205688 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.205409 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmg2g\" (UniqueName: \"kubernetes.io/projected/c6f478f4-6ae4-4ed8-8881-b1945228d966-kube-api-access-jmg2g\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:41:53.791903 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.791869 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6944cc9ff8-gkv7x"] Sep 11 14:41:53.792271 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.792115 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6f478f4-6ae4-4ed8-8881-b1945228d966" containerName="registry" Sep 11 14:41:53.792271 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.792126 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6f478f4-6ae4-4ed8-8881-b1945228d966" containerName="registry" Sep 11 14:41:53.792271 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.792170 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6f478f4-6ae4-4ed8-8881-b1945228d966" containerName="registry" Sep 11 14:41:53.794769 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.794751 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" Sep 11 14:41:53.797784 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.797755 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 11 14:41:53.797784 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.797758 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 11 14:41:53.803556 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.799240 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1mb3lbmc1kvjf\"" Sep 11 14:41:53.803556 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.799496 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2j7mj\"" Sep 11 14:41:53.803556 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.799916 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 11 14:41:53.803556 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.800267 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 11 14:41:53.806646 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.806623 2583 generic.go:356] "Generic (PLEG): container finished" podID="c6f478f4-6ae4-4ed8-8881-b1945228d966" containerID="9b702ad63e4376a980e5067920e10d6f100787536b63886b2a6997b38f60cb76" exitCode=0 Sep 11 14:41:53.806779 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.806672 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" Sep 11 14:41:53.806779 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.806739 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" event={"ID":"c6f478f4-6ae4-4ed8-8881-b1945228d966","Type":"ContainerDied","Data":"9b702ad63e4376a980e5067920e10d6f100787536b63886b2a6997b38f60cb76"} Sep 11 14:41:53.806899 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.806827 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56f4498d5f-4dwl8" event={"ID":"c6f478f4-6ae4-4ed8-8881-b1945228d966","Type":"ContainerDied","Data":"662ab23cff42a14b81fc3ec1de600a3e6d1382ebd98941c37e554d8e73cf7ffb"} Sep 11 14:41:53.806899 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.806859 2583 scope.go:117] "RemoveContainer" containerID="9b702ad63e4376a980e5067920e10d6f100787536b63886b2a6997b38f60cb76" Sep 11 14:41:53.809269 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.809252 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fp8dm" event={"ID":"c8391c39-3072-4cba-9812-5dc45d82589d","Type":"ContainerStarted","Data":"e6130485327a22a5b3c432043623fa16861e1ac93fa12b8af0c04f492a2c0aa0"} Sep 11 14:41:53.809351 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.809294 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fp8dm" event={"ID":"c8391c39-3072-4cba-9812-5dc45d82589d","Type":"ContainerStarted","Data":"e29619453af45bb34184e401f6e8589bd2ab6ddb85c70e3f851325c943234543"} Sep 11 14:41:53.811997 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.811976 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6944cc9ff8-gkv7x"] Sep 11 14:41:53.816276 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.816261 2583 scope.go:117] "RemoveContainer" containerID="9b702ad63e4376a980e5067920e10d6f100787536b63886b2a6997b38f60cb76" Sep 11 14:41:53.816518 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:41:53.816497 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b702ad63e4376a980e5067920e10d6f100787536b63886b2a6997b38f60cb76\": container with ID starting with 9b702ad63e4376a980e5067920e10d6f100787536b63886b2a6997b38f60cb76 not found: ID does not exist" containerID="9b702ad63e4376a980e5067920e10d6f100787536b63886b2a6997b38f60cb76" Sep 11 14:41:53.816574 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.816524 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b702ad63e4376a980e5067920e10d6f100787536b63886b2a6997b38f60cb76"} err="failed to get container status \"9b702ad63e4376a980e5067920e10d6f100787536b63886b2a6997b38f60cb76\": rpc error: code = NotFound desc = could not find container \"9b702ad63e4376a980e5067920e10d6f100787536b63886b2a6997b38f60cb76\": container with ID starting with 9b702ad63e4376a980e5067920e10d6f100787536b63886b2a6997b38f60cb76 not found: ID does not exist" Sep 11 14:41:53.844303 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.844279 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56f4498d5f-4dwl8"] Sep 11 14:41:53.848651 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.848634 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-56f4498d5f-4dwl8"] Sep 11 14:41:53.867700 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.867658 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fp8dm" podStartSLOduration=3.41315114 podStartE2EDuration="4.867643575s" totalImagesPullingTime="1.375144365s" totalInitContainerRuntime="79.34807ms" isStatefulPod=true podCreationTimestamp="2026-09-11 14:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:53.867145765 +0000 UTC m=+105.111131423" watchObservedRunningTime="2026-09-11 14:41:53.867643575 +0000 UTC m=+105.111629241" Sep 11 14:41:53.911184 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.911164 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a8454816-3d65-4a4a-bb36-8b482cca02c3-secret-metrics-server-client-certs\") pod \"metrics-server-6944cc9ff8-gkv7x\" (UID: \"a8454816-3d65-4a4a-bb36-8b482cca02c3\") " pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" Sep 11 14:41:53.911267 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.911198 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a8454816-3d65-4a4a-bb36-8b482cca02c3-metrics-server-audit-profiles\") pod \"metrics-server-6944cc9ff8-gkv7x\" (UID: \"a8454816-3d65-4a4a-bb36-8b482cca02c3\") " pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" Sep 11 14:41:53.911267 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.911228 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q7km\" (UniqueName: \"kubernetes.io/projected/a8454816-3d65-4a4a-bb36-8b482cca02c3-kube-api-access-4q7km\") pod \"metrics-server-6944cc9ff8-gkv7x\" (UID: \"a8454816-3d65-4a4a-bb36-8b482cca02c3\") " pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" Sep 11 14:41:53.911267 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.911258 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a8454816-3d65-4a4a-bb36-8b482cca02c3-client-ca-bundle\") pod \"metrics-server-6944cc9ff8-gkv7x\" (UID: \"a8454816-3d65-4a4a-bb36-8b482cca02c3\") " pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" Sep 11 14:41:53.911363 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.911278 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a8454816-3d65-4a4a-bb36-8b482cca02c3-secret-metrics-server-tls\") pod \"metrics-server-6944cc9ff8-gkv7x\" (UID: \"a8454816-3d65-4a4a-bb36-8b482cca02c3\") " pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" Sep 11 14:41:53.911363 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.911316 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a8454816-3d65-4a4a-bb36-8b482cca02c3-audit-log\") pod \"metrics-server-6944cc9ff8-gkv7x\" (UID: \"a8454816-3d65-4a4a-bb36-8b482cca02c3\") " pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" Sep 11 14:41:53.911363 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:53.911335 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8454816-3d65-4a4a-bb36-8b482cca02c3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6944cc9ff8-gkv7x\" (UID: \"a8454816-3d65-4a4a-bb36-8b482cca02c3\") " pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" Sep 11 14:41:54.012880 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.012856 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a8454816-3d65-4a4a-bb36-8b482cca02c3-secret-metrics-server-client-certs\") pod \"metrics-server-6944cc9ff8-gkv7x\" (UID: \"a8454816-3d65-4a4a-bb36-8b482cca02c3\") " pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" Sep 11 14:41:54.012946 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.012889 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a8454816-3d65-4a4a-bb36-8b482cca02c3-metrics-server-audit-profiles\") pod \"metrics-server-6944cc9ff8-gkv7x\" (UID: \"a8454816-3d65-4a4a-bb36-8b482cca02c3\") " pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" Sep 11 14:41:54.012946 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.012907 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q7km\" (UniqueName: \"kubernetes.io/projected/a8454816-3d65-4a4a-bb36-8b482cca02c3-kube-api-access-4q7km\") pod \"metrics-server-6944cc9ff8-gkv7x\" (UID: \"a8454816-3d65-4a4a-bb36-8b482cca02c3\") " pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" Sep 11 14:41:54.012946 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.012928 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a8454816-3d65-4a4a-bb36-8b482cca02c3-client-ca-bundle\") pod \"metrics-server-6944cc9ff8-gkv7x\" (UID: \"a8454816-3d65-4a4a-bb36-8b482cca02c3\") " pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" Sep 11 14:41:54.013104 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.012957 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a8454816-3d65-4a4a-bb36-8b482cca02c3-secret-metrics-server-tls\") pod \"metrics-server-6944cc9ff8-gkv7x\" (UID: \"a8454816-3d65-4a4a-bb36-8b482cca02c3\") " pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" Sep 11 14:41:54.013104 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.012987 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a8454816-3d65-4a4a-bb36-8b482cca02c3-audit-log\") pod \"metrics-server-6944cc9ff8-gkv7x\" (UID: \"a8454816-3d65-4a4a-bb36-8b482cca02c3\") " pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" Sep 11 14:41:54.013104 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.013017 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8454816-3d65-4a4a-bb36-8b482cca02c3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6944cc9ff8-gkv7x\" (UID: \"a8454816-3d65-4a4a-bb36-8b482cca02c3\") " pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" Sep 11 14:41:54.013382 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.013358 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a8454816-3d65-4a4a-bb36-8b482cca02c3-audit-log\") pod \"metrics-server-6944cc9ff8-gkv7x\" (UID: \"a8454816-3d65-4a4a-bb36-8b482cca02c3\") " pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" Sep 11 14:41:54.013721 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.013688 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8454816-3d65-4a4a-bb36-8b482cca02c3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6944cc9ff8-gkv7x\" (UID: \"a8454816-3d65-4a4a-bb36-8b482cca02c3\") " pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" Sep 11 14:41:54.013922 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.013906 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a8454816-3d65-4a4a-bb36-8b482cca02c3-metrics-server-audit-profiles\") pod \"metrics-server-6944cc9ff8-gkv7x\" (UID: \"a8454816-3d65-4a4a-bb36-8b482cca02c3\") " pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" Sep 11 14:41:54.015305 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.015283 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a8454816-3d65-4a4a-bb36-8b482cca02c3-secret-metrics-server-tls\") pod \"metrics-server-6944cc9ff8-gkv7x\" (UID: \"a8454816-3d65-4a4a-bb36-8b482cca02c3\") " pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" Sep 11 14:41:54.015689 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.015669 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a8454816-3d65-4a4a-bb36-8b482cca02c3-client-ca-bundle\") pod \"metrics-server-6944cc9ff8-gkv7x\" (UID: \"a8454816-3d65-4a4a-bb36-8b482cca02c3\") " pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" Sep 11 14:41:54.015792 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.015773 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a8454816-3d65-4a4a-bb36-8b482cca02c3-secret-metrics-server-client-certs\") pod \"metrics-server-6944cc9ff8-gkv7x\" (UID: \"a8454816-3d65-4a4a-bb36-8b482cca02c3\") " pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" Sep 11 14:41:54.022009 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.021988 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q7km\" (UniqueName: \"kubernetes.io/projected/a8454816-3d65-4a4a-bb36-8b482cca02c3-kube-api-access-4q7km\") pod \"metrics-server-6944cc9ff8-gkv7x\" (UID: \"a8454816-3d65-4a4a-bb36-8b482cca02c3\") " pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" Sep 11 14:41:54.105801 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.105740 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" Sep 11 14:41:54.169687 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.169343 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bf98f8d86-pvbhm"] Sep 11 14:41:54.172802 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.172768 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf98f8d86-pvbhm" Sep 11 14:41:54.175824 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.175801 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 11 14:41:54.176048 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.176030 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 11 14:41:54.176138 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.176122 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 11 14:41:54.176211 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.176138 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8v4w7\"" Sep 11 14:41:54.176268 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.176231 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 11 14:41:54.176359 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.176347 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 11 14:41:54.176948 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.176929 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 11 14:41:54.177481 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.177358 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 11 14:41:54.179508 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.179488 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bf98f8d86-pvbhm"] Sep 11 14:41:54.183526 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.183349 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 11 14:41:54.315839 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.315809 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gnlt\" (UniqueName: \"kubernetes.io/projected/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-kube-api-access-9gnlt\") pod \"console-7bf98f8d86-pvbhm\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " pod="openshift-console/console-7bf98f8d86-pvbhm" Sep 11 14:41:54.315996 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.315848 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-oauth-serving-cert\") pod \"console-7bf98f8d86-pvbhm\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " pod="openshift-console/console-7bf98f8d86-pvbhm" Sep 11 14:41:54.315996 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.315871 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-trusted-ca-bundle\") pod \"console-7bf98f8d86-pvbhm\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " pod="openshift-console/console-7bf98f8d86-pvbhm" Sep 11 14:41:54.315996 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.315963 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-service-ca\") pod \"console-7bf98f8d86-pvbhm\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " pod="openshift-console/console-7bf98f8d86-pvbhm" Sep 11 14:41:54.316118 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.316012 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-console-config\") pod \"console-7bf98f8d86-pvbhm\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " pod="openshift-console/console-7bf98f8d86-pvbhm" Sep 11 14:41:54.316118 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.316031 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-console-serving-cert\") pod \"console-7bf98f8d86-pvbhm\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " pod="openshift-console/console-7bf98f8d86-pvbhm" Sep 11 14:41:54.316118 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.316045 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-console-oauth-config\") pod \"console-7bf98f8d86-pvbhm\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " pod="openshift-console/console-7bf98f8d86-pvbhm" Sep 11 14:41:54.380546 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.380521 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6944cc9ff8-gkv7x"] Sep 11 14:41:54.383411 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:41:54.383388 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8454816_3d65_4a4a_bb36_8b482cca02c3.slice/crio-9566a5794f46053661f9c18a7874b5e7e6bf5227bd980366eca7b2815874948a WatchSource:0}: Error finding container 9566a5794f46053661f9c18a7874b5e7e6bf5227bd980366eca7b2815874948a: Status 404 returned error can't find the container with id 9566a5794f46053661f9c18a7874b5e7e6bf5227bd980366eca7b2815874948a Sep 11 14:41:54.416539 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.416520 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-oauth-serving-cert\") pod \"console-7bf98f8d86-pvbhm\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " pod="openshift-console/console-7bf98f8d86-pvbhm" Sep 11 14:41:54.416638 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.416549 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-trusted-ca-bundle\") pod \"console-7bf98f8d86-pvbhm\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " pod="openshift-console/console-7bf98f8d86-pvbhm" Sep 11 14:41:54.416638 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.416582 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-service-ca\") pod \"console-7bf98f8d86-pvbhm\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " pod="openshift-console/console-7bf98f8d86-pvbhm" Sep 11 14:41:54.416743 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.416650 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-console-config\") pod \"console-7bf98f8d86-pvbhm\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " pod="openshift-console/console-7bf98f8d86-pvbhm" Sep 11 14:41:54.416743 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.416679 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-console-serving-cert\") pod \"console-7bf98f8d86-pvbhm\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " pod="openshift-console/console-7bf98f8d86-pvbhm" Sep 11 14:41:54.416743 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.416721 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-console-oauth-config\") pod \"console-7bf98f8d86-pvbhm\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " pod="openshift-console/console-7bf98f8d86-pvbhm" Sep 11 14:41:54.416871 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.416760 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gnlt\" (UniqueName: \"kubernetes.io/projected/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-kube-api-access-9gnlt\") pod \"console-7bf98f8d86-pvbhm\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " pod="openshift-console/console-7bf98f8d86-pvbhm" Sep 11 14:41:54.417336 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.417298 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-oauth-serving-cert\") pod \"console-7bf98f8d86-pvbhm\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " pod="openshift-console/console-7bf98f8d86-pvbhm" Sep 11 14:41:54.417438 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.417363 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-service-ca\") pod \"console-7bf98f8d86-pvbhm\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " pod="openshift-console/console-7bf98f8d86-pvbhm" Sep 11 14:41:54.417438 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.417407 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-console-config\") pod \"console-7bf98f8d86-pvbhm\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " pod="openshift-console/console-7bf98f8d86-pvbhm" Sep 11 14:41:54.417576 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.417557 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-trusted-ca-bundle\") pod \"console-7bf98f8d86-pvbhm\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " pod="openshift-console/console-7bf98f8d86-pvbhm" Sep 11 14:41:54.419184 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.419151 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-console-oauth-config\") pod \"console-7bf98f8d86-pvbhm\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " pod="openshift-console/console-7bf98f8d86-pvbhm" Sep 11 14:41:54.419255 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.419237 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-console-serving-cert\") pod \"console-7bf98f8d86-pvbhm\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " pod="openshift-console/console-7bf98f8d86-pvbhm" Sep 11 14:41:54.424593 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.424577 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gnlt\" (UniqueName: \"kubernetes.io/projected/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-kube-api-access-9gnlt\") pod \"console-7bf98f8d86-pvbhm\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " pod="openshift-console/console-7bf98f8d86-pvbhm" Sep 11 14:41:54.486071 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.486048 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf98f8d86-pvbhm" Sep 11 14:41:54.755662 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.755587 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bf98f8d86-pvbhm"] Sep 11 14:41:54.758876 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:41:54.758854 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f0b750_c1c4_4aa9_af24_5c58fca9fd70.slice/crio-e75afbb1967b47572b8d386fa59eb5edbb29a881fb5cd764aa1e8d891e3bd3e4 WatchSource:0}: Error finding container e75afbb1967b47572b8d386fa59eb5edbb29a881fb5cd764aa1e8d891e3bd3e4: Status 404 returned error can't find the container with id e75afbb1967b47572b8d386fa59eb5edbb29a881fb5cd764aa1e8d891e3bd3e4 Sep 11 14:41:54.813777 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.813741 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf98f8d86-pvbhm" event={"ID":"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70","Type":"ContainerStarted","Data":"e75afbb1967b47572b8d386fa59eb5edbb29a881fb5cd764aa1e8d891e3bd3e4"} Sep 11 14:41:54.815017 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:54.814991 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" event={"ID":"a8454816-3d65-4a4a-bb36-8b482cca02c3","Type":"ContainerStarted","Data":"9566a5794f46053661f9c18a7874b5e7e6bf5227bd980366eca7b2815874948a"} Sep 11 14:41:55.331240 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:55.331206 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6f478f4-6ae4-4ed8-8881-b1945228d966" path="/var/lib/kubelet/pods/c6f478f4-6ae4-4ed8-8881-b1945228d966/volumes" Sep 11 14:41:56.829044 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:56.828965 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" event={"ID":"a8454816-3d65-4a4a-bb36-8b482cca02c3","Type":"ContainerStarted","Data":"505ee61892ea602d2ad854cc79de0cf5c8c3bff92591b20fb3fcde33fe8d1b62"} Sep 11 14:41:56.849060 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:56.849010 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" podStartSLOduration=2.414576656 podStartE2EDuration="3.848990344s" totalImagesPullingTime="1.434413688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:56.848244176 +0000 UTC m=+108.092229835" watchObservedRunningTime="2026-09-11 14:41:56.848990344 +0000 UTC m=+108.092976004" Sep 11 14:41:57.833766 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:57.833727 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf98f8d86-pvbhm" event={"ID":"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70","Type":"ContainerStarted","Data":"c98e281a89615813d716fbe84fe3e5661c043a3a8e19ecde3cc06f7d6f0bcd5a"} Sep 11 14:41:57.854375 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:57.854333 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bf98f8d86-pvbhm" podStartSLOduration=1.248433446 podStartE2EDuration="3.854318031s" totalImagesPullingTime="2.605884585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:57.852761657 +0000 UTC m=+109.096747308" watchObservedRunningTime="2026-09-11 14:41:57.854318031 +0000 UTC m=+109.098303691" Sep 11 14:41:57.936017 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:41:57.935988 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bf98f8d86-pvbhm"] Sep 11 14:42:04.486613 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:04.486582 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bf98f8d86-pvbhm" Sep 11 14:42:14.106761 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:14.106680 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" Sep 11 14:42:14.107149 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:14.106789 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" Sep 11 14:42:24.859427 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:24.859385 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bf98f8d86-pvbhm" podUID="d2f0b750-c1c4-4aa9-af24-5c58fca9fd70" containerName="console" containerID="cri-o://c98e281a89615813d716fbe84fe3e5661c043a3a8e19ecde3cc06f7d6f0bcd5a" gracePeriod=15 Sep 11 14:42:25.090018 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.089995 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf98f8d86-pvbhm" Sep 11 14:42:25.134985 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.134959 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gnlt\" (UniqueName: \"kubernetes.io/projected/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-kube-api-access-9gnlt\") pod \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " Sep 11 14:42:25.135127 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.134996 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-trusted-ca-bundle\") pod \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " Sep 11 14:42:25.135127 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.135016 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-console-oauth-config\") pod \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " Sep 11 14:42:25.135127 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.135036 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-console-config\") pod \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " Sep 11 14:42:25.135127 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.135058 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-service-ca\") pod \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " Sep 11 14:42:25.135127 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.135084 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-console-serving-cert\") pod \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " Sep 11 14:42:25.135372 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.135150 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-oauth-serving-cert\") pod \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\" (UID: \"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70\") " Sep 11 14:42:25.135569 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.135538 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d2f0b750-c1c4-4aa9-af24-5c58fca9fd70" (UID: "d2f0b750-c1c4-4aa9-af24-5c58fca9fd70"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:42:25.135652 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.135538 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-console-config" (OuterVolumeSpecName: "console-config") pod "d2f0b750-c1c4-4aa9-af24-5c58fca9fd70" (UID: "d2f0b750-c1c4-4aa9-af24-5c58fca9fd70"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:42:25.135652 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.135539 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-service-ca" (OuterVolumeSpecName: "service-ca") pod "d2f0b750-c1c4-4aa9-af24-5c58fca9fd70" (UID: "d2f0b750-c1c4-4aa9-af24-5c58fca9fd70"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:42:25.135652 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.135605 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d2f0b750-c1c4-4aa9-af24-5c58fca9fd70" (UID: "d2f0b750-c1c4-4aa9-af24-5c58fca9fd70"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:42:25.137470 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.137445 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d2f0b750-c1c4-4aa9-af24-5c58fca9fd70" (UID: "d2f0b750-c1c4-4aa9-af24-5c58fca9fd70"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:42:25.137571 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.137550 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d2f0b750-c1c4-4aa9-af24-5c58fca9fd70" (UID: "d2f0b750-c1c4-4aa9-af24-5c58fca9fd70"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:42:25.137609 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.137577 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-kube-api-access-9gnlt" (OuterVolumeSpecName: "kube-api-access-9gnlt") pod "d2f0b750-c1c4-4aa9-af24-5c58fca9fd70" (UID: "d2f0b750-c1c4-4aa9-af24-5c58fca9fd70"). InnerVolumeSpecName "kube-api-access-9gnlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:42:25.235723 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.235668 2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-oauth-serving-cert\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:42:25.235723 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.235724 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gnlt\" (UniqueName: \"kubernetes.io/projected/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-kube-api-access-9gnlt\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:42:25.235918 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.235741 2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-trusted-ca-bundle\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:42:25.235918 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.235754 2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-console-oauth-config\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:42:25.235918 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.235764 2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-console-config\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:42:25.235918 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.235772 2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-service-ca\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:42:25.235918 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.235779 2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70-console-serving-cert\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:42:25.924954 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.924918 2583 generic.go:356] "Generic (PLEG): container finished" podID="d2f0b750-c1c4-4aa9-af24-5c58fca9fd70" containerID="c98e281a89615813d716fbe84fe3e5661c043a3a8e19ecde3cc06f7d6f0bcd5a" exitCode=0 Sep 11 14:42:25.924954 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.924958 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf98f8d86-pvbhm" event={"ID":"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70","Type":"ContainerDied","Data":"c98e281a89615813d716fbe84fe3e5661c043a3a8e19ecde3cc06f7d6f0bcd5a"} Sep 11 14:42:25.925440 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.925002 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf98f8d86-pvbhm" event={"ID":"d2f0b750-c1c4-4aa9-af24-5c58fca9fd70","Type":"ContainerDied","Data":"e75afbb1967b47572b8d386fa59eb5edbb29a881fb5cd764aa1e8d891e3bd3e4"} Sep 11 14:42:25.925440 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.925028 2583 scope.go:117] "RemoveContainer" containerID="c98e281a89615813d716fbe84fe3e5661c043a3a8e19ecde3cc06f7d6f0bcd5a" Sep 11 14:42:25.925440 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.924961 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf98f8d86-pvbhm" Sep 11 14:42:25.932728 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.932692 2583 scope.go:117] "RemoveContainer" containerID="c98e281a89615813d716fbe84fe3e5661c043a3a8e19ecde3cc06f7d6f0bcd5a" Sep 11 14:42:25.933020 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:42:25.933000 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c98e281a89615813d716fbe84fe3e5661c043a3a8e19ecde3cc06f7d6f0bcd5a\": container with ID starting with c98e281a89615813d716fbe84fe3e5661c043a3a8e19ecde3cc06f7d6f0bcd5a not found: ID does not exist" containerID="c98e281a89615813d716fbe84fe3e5661c043a3a8e19ecde3cc06f7d6f0bcd5a" Sep 11 14:42:25.933079 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.933027 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c98e281a89615813d716fbe84fe3e5661c043a3a8e19ecde3cc06f7d6f0bcd5a"} err="failed to get container status \"c98e281a89615813d716fbe84fe3e5661c043a3a8e19ecde3cc06f7d6f0bcd5a\": rpc error: code = NotFound desc = could not find container \"c98e281a89615813d716fbe84fe3e5661c043a3a8e19ecde3cc06f7d6f0bcd5a\": container with ID starting with c98e281a89615813d716fbe84fe3e5661c043a3a8e19ecde3cc06f7d6f0bcd5a not found: ID does not exist" Sep 11 14:42:25.941812 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.941791 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bf98f8d86-pvbhm"] Sep 11 14:42:25.947806 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:25.947786 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bf98f8d86-pvbhm"] Sep 11 14:42:27.329542 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:27.329513 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2f0b750-c1c4-4aa9-af24-5c58fca9fd70" path="/var/lib/kubelet/pods/d2f0b750-c1c4-4aa9-af24-5c58fca9fd70/volumes" Sep 11 14:42:34.112039 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:34.112005 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" Sep 11 14:42:34.116951 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:42:34.116925 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6944cc9ff8-gkv7x" Sep 11 14:43:14.052776 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.052737 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bc9d7ddcc-zzvvs"] Sep 11 14:43:14.053157 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.052990 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f0b750-c1c4-4aa9-af24-5c58fca9fd70" containerName="console" Sep 11 14:43:14.053157 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.053005 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f0b750-c1c4-4aa9-af24-5c58fca9fd70" containerName="console" Sep 11 14:43:14.053157 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.053056 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2f0b750-c1c4-4aa9-af24-5c58fca9fd70" containerName="console" Sep 11 14:43:14.059093 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.059065 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:43:14.061726 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.061681 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 11 14:43:14.062654 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.062636 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 11 14:43:14.062654 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.062645 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8v4w7\"" Sep 11 14:43:14.062823 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.062807 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 11 14:43:14.063085 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.063067 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 11 14:43:14.063908 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.063892 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 11 14:43:14.064074 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.064059 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 11 14:43:14.064114 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.064071 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 11 14:43:14.066840 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.066822 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 11 14:43:14.072801 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.072780 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bc9d7ddcc-zzvvs"] Sep 11 14:43:14.161039 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.161016 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8343026f-e77a-4c2e-b224-99e9cefd5ffa-service-ca\") pod \"console-bc9d7ddcc-zzvvs\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:43:14.161184 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.161057 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8343026f-e77a-4c2e-b224-99e9cefd5ffa-oauth-serving-cert\") pod \"console-bc9d7ddcc-zzvvs\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:43:14.161184 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.161077 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8343026f-e77a-4c2e-b224-99e9cefd5ffa-trusted-ca-bundle\") pod \"console-bc9d7ddcc-zzvvs\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:43:14.161184 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.161143 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8343026f-e77a-4c2e-b224-99e9cefd5ffa-console-config\") pod \"console-bc9d7ddcc-zzvvs\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:43:14.161184 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.161163 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8343026f-e77a-4c2e-b224-99e9cefd5ffa-console-serving-cert\") pod \"console-bc9d7ddcc-zzvvs\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:43:14.161184 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.161179 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8343026f-e77a-4c2e-b224-99e9cefd5ffa-console-oauth-config\") pod \"console-bc9d7ddcc-zzvvs\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:43:14.161364 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.161226 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mmlz\" (UniqueName: \"kubernetes.io/projected/8343026f-e77a-4c2e-b224-99e9cefd5ffa-kube-api-access-7mmlz\") pod \"console-bc9d7ddcc-zzvvs\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:43:14.262532 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.262502 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mmlz\" (UniqueName: \"kubernetes.io/projected/8343026f-e77a-4c2e-b224-99e9cefd5ffa-kube-api-access-7mmlz\") pod \"console-bc9d7ddcc-zzvvs\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:43:14.262691 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.262540 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8343026f-e77a-4c2e-b224-99e9cefd5ffa-service-ca\") pod \"console-bc9d7ddcc-zzvvs\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:43:14.262691 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.262571 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8343026f-e77a-4c2e-b224-99e9cefd5ffa-oauth-serving-cert\") pod \"console-bc9d7ddcc-zzvvs\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:43:14.262691 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.262589 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8343026f-e77a-4c2e-b224-99e9cefd5ffa-trusted-ca-bundle\") pod \"console-bc9d7ddcc-zzvvs\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:43:14.262691 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.262617 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8343026f-e77a-4c2e-b224-99e9cefd5ffa-console-config\") pod \"console-bc9d7ddcc-zzvvs\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:43:14.262691 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.262634 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8343026f-e77a-4c2e-b224-99e9cefd5ffa-console-serving-cert\") pod \"console-bc9d7ddcc-zzvvs\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:43:14.262691 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.262650 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8343026f-e77a-4c2e-b224-99e9cefd5ffa-console-oauth-config\") pod \"console-bc9d7ddcc-zzvvs\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:43:14.263314 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.263290 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8343026f-e77a-4c2e-b224-99e9cefd5ffa-oauth-serving-cert\") pod \"console-bc9d7ddcc-zzvvs\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:43:14.263390 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.263329 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8343026f-e77a-4c2e-b224-99e9cefd5ffa-service-ca\") pod \"console-bc9d7ddcc-zzvvs\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:43:14.263440 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.263393 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8343026f-e77a-4c2e-b224-99e9cefd5ffa-console-config\") pod \"console-bc9d7ddcc-zzvvs\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:43:14.263528 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.263512 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8343026f-e77a-4c2e-b224-99e9cefd5ffa-trusted-ca-bundle\") pod \"console-bc9d7ddcc-zzvvs\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:43:14.265092 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.265071 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8343026f-e77a-4c2e-b224-99e9cefd5ffa-console-serving-cert\") pod \"console-bc9d7ddcc-zzvvs\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:43:14.265219 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.265199 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8343026f-e77a-4c2e-b224-99e9cefd5ffa-console-oauth-config\") pod \"console-bc9d7ddcc-zzvvs\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:43:14.272296 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.272269 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mmlz\" (UniqueName: \"kubernetes.io/projected/8343026f-e77a-4c2e-b224-99e9cefd5ffa-kube-api-access-7mmlz\") pod \"console-bc9d7ddcc-zzvvs\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:43:14.368796 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.368718 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:43:14.644978 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:14.644949 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bc9d7ddcc-zzvvs"] Sep 11 14:43:14.648067 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:43:14.648042 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8343026f_e77a_4c2e_b224_99e9cefd5ffa.slice/crio-42bb6fe5a0950f4e02aa9355445f15b97f04fef3f37de1fd234a6fec4332fa59 WatchSource:0}: Error finding container 42bb6fe5a0950f4e02aa9355445f15b97f04fef3f37de1fd234a6fec4332fa59: Status 404 returned error can't find the container with id 42bb6fe5a0950f4e02aa9355445f15b97f04fef3f37de1fd234a6fec4332fa59 Sep 11 14:43:15.090906 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:15.090809 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bc9d7ddcc-zzvvs" event={"ID":"8343026f-e77a-4c2e-b224-99e9cefd5ffa","Type":"ContainerStarted","Data":"4d887057c99e6fc189f757349396c5f2fe35b08dc52e196732bb266f4f32731e"} Sep 11 14:43:15.090906 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:15.090866 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bc9d7ddcc-zzvvs" event={"ID":"8343026f-e77a-4c2e-b224-99e9cefd5ffa","Type":"ContainerStarted","Data":"42bb6fe5a0950f4e02aa9355445f15b97f04fef3f37de1fd234a6fec4332fa59"} Sep 11 14:43:15.113555 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:15.113508 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bc9d7ddcc-zzvvs" podStartSLOduration=1.113491002 podStartE2EDuration="1.113491002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:43:15.112345628 +0000 UTC m=+186.356331285" watchObservedRunningTime="2026-09-11 14:43:15.113491002 +0000 UTC m=+186.357476659" Sep 11 14:43:24.368886 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:24.368849 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:43:24.369350 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:24.368909 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:43:24.373577 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:24.373551 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:43:25.125786 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:43:25.125760 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:44:16.627887 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:16.627851 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9"] Sep 11 14:44:16.631092 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:16.631072 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9" Sep 11 14:44:16.633472 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:16.633448 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lvjqc\"" Sep 11 14:44:16.633582 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:16.633563 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 14:44:16.634222 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:16.634206 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 14:44:16.643766 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:16.643745 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9"] Sep 11 14:44:16.768765 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:16.768729 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kxxh\" (UniqueName: \"kubernetes.io/projected/fc971292-c8f3-4e03-b4e5-83a0f48611d8-kube-api-access-7kxxh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9\" (UID: \"fc971292-c8f3-4e03-b4e5-83a0f48611d8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9" Sep 11 14:44:16.768765 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:16.768770 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc971292-c8f3-4e03-b4e5-83a0f48611d8-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9\" (UID: \"fc971292-c8f3-4e03-b4e5-83a0f48611d8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9" Sep 11 14:44:16.768963 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:16.768869 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc971292-c8f3-4e03-b4e5-83a0f48611d8-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9\" (UID: \"fc971292-c8f3-4e03-b4e5-83a0f48611d8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9" Sep 11 14:44:16.869611 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:16.869576 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc971292-c8f3-4e03-b4e5-83a0f48611d8-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9\" (UID: \"fc971292-c8f3-4e03-b4e5-83a0f48611d8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9" Sep 11 14:44:16.869808 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:16.869630 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kxxh\" (UniqueName: \"kubernetes.io/projected/fc971292-c8f3-4e03-b4e5-83a0f48611d8-kube-api-access-7kxxh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9\" (UID: \"fc971292-c8f3-4e03-b4e5-83a0f48611d8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9" Sep 11 14:44:16.869808 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:16.869754 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc971292-c8f3-4e03-b4e5-83a0f48611d8-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9\" (UID: \"fc971292-c8f3-4e03-b4e5-83a0f48611d8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9" Sep 11 14:44:16.870058 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:16.870036 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc971292-c8f3-4e03-b4e5-83a0f48611d8-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9\" (UID: \"fc971292-c8f3-4e03-b4e5-83a0f48611d8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9" Sep 11 14:44:16.870122 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:16.870062 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc971292-c8f3-4e03-b4e5-83a0f48611d8-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9\" (UID: \"fc971292-c8f3-4e03-b4e5-83a0f48611d8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9" Sep 11 14:44:16.881100 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:16.881054 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kxxh\" (UniqueName: \"kubernetes.io/projected/fc971292-c8f3-4e03-b4e5-83a0f48611d8-kube-api-access-7kxxh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9\" (UID: \"fc971292-c8f3-4e03-b4e5-83a0f48611d8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9" Sep 11 14:44:16.939905 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:16.939884 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9" Sep 11 14:44:17.208239 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:17.208163 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9"] Sep 11 14:44:17.211028 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:44:17.211002 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc971292_c8f3_4e03_b4e5_83a0f48611d8.slice/crio-4a8f5a797cd04eba3f1cd2e9a4d4d211ed711fc7cebe593528079cf265f72f6a WatchSource:0}: Error finding container 4a8f5a797cd04eba3f1cd2e9a4d4d211ed711fc7cebe593528079cf265f72f6a: Status 404 returned error can't find the container with id 4a8f5a797cd04eba3f1cd2e9a4d4d211ed711fc7cebe593528079cf265f72f6a Sep 11 14:44:17.282915 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:17.282887 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9" event={"ID":"fc971292-c8f3-4e03-b4e5-83a0f48611d8","Type":"ContainerStarted","Data":"4a8f5a797cd04eba3f1cd2e9a4d4d211ed711fc7cebe593528079cf265f72f6a"} Sep 11 14:44:24.311753 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:24.311686 2583 generic.go:356] "Generic (PLEG): container finished" podID="fc971292-c8f3-4e03-b4e5-83a0f48611d8" containerID="a327d8ab145d9f6ba8450a23f57c57cb173d523e097e4f73fe8ebe928d88b1f4" exitCode=0 Sep 11 14:44:24.312134 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:24.311767 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9" event={"ID":"fc971292-c8f3-4e03-b4e5-83a0f48611d8","Type":"ContainerDied","Data":"a327d8ab145d9f6ba8450a23f57c57cb173d523e097e4f73fe8ebe928d88b1f4"} Sep 11 14:44:26.319550 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:26.319514 2583 generic.go:356] "Generic (PLEG): container finished" podID="fc971292-c8f3-4e03-b4e5-83a0f48611d8" containerID="885a744f7fe7f8d001e940750b86171faee6233eaafb6f780bf9e741a6e05108" exitCode=0 Sep 11 14:44:26.319902 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:26.319598 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9" event={"ID":"fc971292-c8f3-4e03-b4e5-83a0f48611d8","Type":"ContainerDied","Data":"885a744f7fe7f8d001e940750b86171faee6233eaafb6f780bf9e741a6e05108"} Sep 11 14:44:34.348131 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:34.348093 2583 generic.go:356] "Generic (PLEG): container finished" podID="fc971292-c8f3-4e03-b4e5-83a0f48611d8" containerID="7c310b8466e77f25694d76a58fc3763f92697c5bc831da6c746945ad5725f5c1" exitCode=0 Sep 11 14:44:34.348484 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:34.348144 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9" event={"ID":"fc971292-c8f3-4e03-b4e5-83a0f48611d8","Type":"ContainerDied","Data":"7c310b8466e77f25694d76a58fc3763f92697c5bc831da6c746945ad5725f5c1"} Sep 11 14:44:35.471601 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:35.471577 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9" Sep 11 14:44:35.500561 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:35.500539 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc971292-c8f3-4e03-b4e5-83a0f48611d8-util\") pod \"fc971292-c8f3-4e03-b4e5-83a0f48611d8\" (UID: \"fc971292-c8f3-4e03-b4e5-83a0f48611d8\") " Sep 11 14:44:35.500666 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:35.500589 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kxxh\" (UniqueName: \"kubernetes.io/projected/fc971292-c8f3-4e03-b4e5-83a0f48611d8-kube-api-access-7kxxh\") pod \"fc971292-c8f3-4e03-b4e5-83a0f48611d8\" (UID: \"fc971292-c8f3-4e03-b4e5-83a0f48611d8\") " Sep 11 14:44:35.500666 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:35.500606 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc971292-c8f3-4e03-b4e5-83a0f48611d8-bundle\") pod \"fc971292-c8f3-4e03-b4e5-83a0f48611d8\" (UID: \"fc971292-c8f3-4e03-b4e5-83a0f48611d8\") " Sep 11 14:44:35.501247 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:35.501221 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc971292-c8f3-4e03-b4e5-83a0f48611d8-bundle" (OuterVolumeSpecName: "bundle") pod "fc971292-c8f3-4e03-b4e5-83a0f48611d8" (UID: "fc971292-c8f3-4e03-b4e5-83a0f48611d8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:35.502672 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:35.502649 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc971292-c8f3-4e03-b4e5-83a0f48611d8-kube-api-access-7kxxh" (OuterVolumeSpecName: "kube-api-access-7kxxh") pod "fc971292-c8f3-4e03-b4e5-83a0f48611d8" (UID: "fc971292-c8f3-4e03-b4e5-83a0f48611d8"). InnerVolumeSpecName "kube-api-access-7kxxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:44:35.504935 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:35.504908 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc971292-c8f3-4e03-b4e5-83a0f48611d8-util" (OuterVolumeSpecName: "util") pod "fc971292-c8f3-4e03-b4e5-83a0f48611d8" (UID: "fc971292-c8f3-4e03-b4e5-83a0f48611d8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:35.601905 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:35.601834 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc971292-c8f3-4e03-b4e5-83a0f48611d8-util\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:44:35.601905 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:35.601859 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kxxh\" (UniqueName: \"kubernetes.io/projected/fc971292-c8f3-4e03-b4e5-83a0f48611d8-kube-api-access-7kxxh\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:44:35.601905 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:35.601869 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc971292-c8f3-4e03-b4e5-83a0f48611d8-bundle\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:44:36.356322 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:36.356287 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9" event={"ID":"fc971292-c8f3-4e03-b4e5-83a0f48611d8","Type":"ContainerDied","Data":"4a8f5a797cd04eba3f1cd2e9a4d4d211ed711fc7cebe593528079cf265f72f6a"} Sep 11 14:44:36.356322 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:36.356328 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a8f5a797cd04eba3f1cd2e9a4d4d211ed711fc7cebe593528079cf265f72f6a" Sep 11 14:44:36.356555 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:36.356329 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7kl9" Sep 11 14:44:39.661615 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:39.661587 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n29h"] Sep 11 14:44:39.661984 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:39.661829 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc971292-c8f3-4e03-b4e5-83a0f48611d8" containerName="util" Sep 11 14:44:39.661984 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:39.661839 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc971292-c8f3-4e03-b4e5-83a0f48611d8" containerName="util" Sep 11 14:44:39.661984 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:39.661847 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc971292-c8f3-4e03-b4e5-83a0f48611d8" containerName="extract" Sep 11 14:44:39.661984 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:39.661852 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc971292-c8f3-4e03-b4e5-83a0f48611d8" containerName="extract" Sep 11 14:44:39.661984 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:39.661863 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc971292-c8f3-4e03-b4e5-83a0f48611d8" containerName="pull" Sep 11 14:44:39.661984 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:39.661868 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc971292-c8f3-4e03-b4e5-83a0f48611d8" containerName="pull" Sep 11 14:44:39.661984 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:39.661909 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc971292-c8f3-4e03-b4e5-83a0f48611d8" containerName="extract" Sep 11 14:44:39.665122 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:39.665105 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n29h" Sep 11 14:44:39.667688 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:39.667668 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-9jf5b\"" Sep 11 14:44:39.667788 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:39.667745 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 11 14:44:39.667865 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:39.667842 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 11 14:44:39.680816 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:39.680798 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n29h"] Sep 11 14:44:39.730136 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:39.730111 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1b75e0d-28fd-425a-8a13-005ff843f0f7-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9n29h\" (UID: \"f1b75e0d-28fd-425a-8a13-005ff843f0f7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n29h" Sep 11 14:44:39.730254 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:39.730160 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jtrc\" (UniqueName: \"kubernetes.io/projected/f1b75e0d-28fd-425a-8a13-005ff843f0f7-kube-api-access-7jtrc\") pod \"cert-manager-operator-controller-manager-55c444bf86-9n29h\" (UID: \"f1b75e0d-28fd-425a-8a13-005ff843f0f7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n29h" Sep 11 14:44:39.831473 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:39.831446 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1b75e0d-28fd-425a-8a13-005ff843f0f7-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9n29h\" (UID: \"f1b75e0d-28fd-425a-8a13-005ff843f0f7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n29h" Sep 11 14:44:39.831569 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:39.831499 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jtrc\" (UniqueName: \"kubernetes.io/projected/f1b75e0d-28fd-425a-8a13-005ff843f0f7-kube-api-access-7jtrc\") pod \"cert-manager-operator-controller-manager-55c444bf86-9n29h\" (UID: \"f1b75e0d-28fd-425a-8a13-005ff843f0f7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n29h" Sep 11 14:44:39.831888 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:39.831870 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1b75e0d-28fd-425a-8a13-005ff843f0f7-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9n29h\" (UID: \"f1b75e0d-28fd-425a-8a13-005ff843f0f7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n29h" Sep 11 14:44:39.840034 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:39.840012 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jtrc\" (UniqueName: \"kubernetes.io/projected/f1b75e0d-28fd-425a-8a13-005ff843f0f7-kube-api-access-7jtrc\") pod \"cert-manager-operator-controller-manager-55c444bf86-9n29h\" (UID: \"f1b75e0d-28fd-425a-8a13-005ff843f0f7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n29h" Sep 11 14:44:39.973975 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:39.973914 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n29h" Sep 11 14:44:40.255422 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:40.255339 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n29h"] Sep 11 14:44:40.259472 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:44:40.259442 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b75e0d_28fd_425a_8a13_005ff843f0f7.slice/crio-589b8d8715360401eb8aa411c138cf802d2c2c7b9a907ac0a07851b2fca0187d WatchSource:0}: Error finding container 589b8d8715360401eb8aa411c138cf802d2c2c7b9a907ac0a07851b2fca0187d: Status 404 returned error can't find the container with id 589b8d8715360401eb8aa411c138cf802d2c2c7b9a907ac0a07851b2fca0187d Sep 11 14:44:40.371121 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:40.371091 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n29h" event={"ID":"f1b75e0d-28fd-425a-8a13-005ff843f0f7","Type":"ContainerStarted","Data":"589b8d8715360401eb8aa411c138cf802d2c2c7b9a907ac0a07851b2fca0187d"} Sep 11 14:44:44.387889 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:44.387851 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n29h" event={"ID":"f1b75e0d-28fd-425a-8a13-005ff843f0f7","Type":"ContainerStarted","Data":"55cc7bdd8ad54f10ca2f1acc853f25ece3caaaf7f0419375c12b7e8607308225"} Sep 11 14:44:44.412202 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:44.411979 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n29h" podStartSLOduration=1.5257044 podStartE2EDuration="5.411961099s" totalImagesPullingTime="3.886256699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:44:44.409362727 +0000 UTC m=+275.653348385" watchObservedRunningTime="2026-09-11 14:44:44.411961099 +0000 UTC m=+275.655946765" Sep 11 14:44:45.213205 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:45.213163 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l"] Sep 11 14:44:45.216395 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:45.216375 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l" Sep 11 14:44:45.219933 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:45.219908 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lvjqc\"" Sep 11 14:44:45.224207 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:45.224188 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 14:44:45.226276 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:45.226260 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 14:44:45.260791 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:45.260769 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l"] Sep 11 14:44:45.268457 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:45.268432 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l\" (UID: \"f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l" Sep 11 14:44:45.268556 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:45.268469 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7czl\" (UniqueName: \"kubernetes.io/projected/f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6-kube-api-access-x7czl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l\" (UID: \"f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l" Sep 11 14:44:45.268556 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:45.268501 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l\" (UID: \"f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l" Sep 11 14:44:45.369739 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:45.369697 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l\" (UID: \"f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l" Sep 11 14:44:45.369823 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:45.369750 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7czl\" (UniqueName: \"kubernetes.io/projected/f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6-kube-api-access-x7czl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l\" (UID: \"f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l" Sep 11 14:44:45.369823 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:45.369778 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l\" (UID: \"f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l" Sep 11 14:44:45.370029 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:45.370013 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l\" (UID: \"f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l" Sep 11 14:44:45.370102 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:45.370082 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l\" (UID: \"f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l" Sep 11 14:44:45.394735 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:45.394689 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7czl\" (UniqueName: \"kubernetes.io/projected/f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6-kube-api-access-x7czl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l\" (UID: \"f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l" Sep 11 14:44:45.524802 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:45.524753 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l" Sep 11 14:44:45.854690 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:44:45.854659 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b0c149_2b5b_4a0d_8b69_58ca1b0df8c6.slice/crio-6f74ba1524cf6674adb0796832ea663e3d5f4d551701b48b0deacef48f8941bc WatchSource:0}: Error finding container 6f74ba1524cf6674adb0796832ea663e3d5f4d551701b48b0deacef48f8941bc: Status 404 returned error can't find the container with id 6f74ba1524cf6674adb0796832ea663e3d5f4d551701b48b0deacef48f8941bc Sep 11 14:44:45.854999 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:45.854978 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l"] Sep 11 14:44:46.397686 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:46.397651 2583 generic.go:356] "Generic (PLEG): container finished" podID="f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6" containerID="064f8c00fe7a68bc731414befb73f67840d5c1dedba9e389d756fab4f3f53549" exitCode=0 Sep 11 14:44:46.398125 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:46.397736 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l" event={"ID":"f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6","Type":"ContainerDied","Data":"064f8c00fe7a68bc731414befb73f67840d5c1dedba9e389d756fab4f3f53549"} Sep 11 14:44:46.398125 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:46.397789 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l" event={"ID":"f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6","Type":"ContainerStarted","Data":"6f74ba1524cf6674adb0796832ea663e3d5f4d551701b48b0deacef48f8941bc"} Sep 11 14:44:48.407270 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:48.407244 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l" event={"ID":"f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6","Type":"ContainerStarted","Data":"960900ca8fe0fc4ddde6c4b3b679e91c022d98cd6f598d2fa64c208f6af2ea4b"} Sep 11 14:44:49.412392 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:49.412354 2583 generic.go:356] "Generic (PLEG): container finished" podID="f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6" containerID="960900ca8fe0fc4ddde6c4b3b679e91c022d98cd6f598d2fa64c208f6af2ea4b" exitCode=0 Sep 11 14:44:49.412392 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:49.412393 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l" event={"ID":"f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6","Type":"ContainerDied","Data":"960900ca8fe0fc4ddde6c4b3b679e91c022d98cd6f598d2fa64c208f6af2ea4b"} Sep 11 14:44:50.418334 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:50.418302 2583 generic.go:356] "Generic (PLEG): container finished" podID="f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6" containerID="19a6ece7929d9c7be6794244a86c02400a4cda49ecca688257f3c39036fc7e4d" exitCode=0 Sep 11 14:44:50.418700 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:50.418389 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l" event={"ID":"f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6","Type":"ContainerDied","Data":"19a6ece7929d9c7be6794244a86c02400a4cda49ecca688257f3c39036fc7e4d"} Sep 11 14:44:51.535644 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:51.535615 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l" Sep 11 14:44:51.615165 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:51.615138 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6-util\") pod \"f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6\" (UID: \"f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6\") " Sep 11 14:44:51.615309 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:51.615182 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6-bundle\") pod \"f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6\" (UID: \"f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6\") " Sep 11 14:44:51.615309 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:51.615214 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7czl\" (UniqueName: \"kubernetes.io/projected/f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6-kube-api-access-x7czl\") pod \"f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6\" (UID: \"f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6\") " Sep 11 14:44:51.615529 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:51.615503 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6-bundle" (OuterVolumeSpecName: "bundle") pod "f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6" (UID: "f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:51.617135 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:51.617113 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6-kube-api-access-x7czl" (OuterVolumeSpecName: "kube-api-access-x7czl") pod "f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6" (UID: "f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6"). InnerVolumeSpecName "kube-api-access-x7czl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:44:51.619602 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:51.619578 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6-util" (OuterVolumeSpecName: "util") pod "f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6" (UID: "f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:51.716088 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:51.716013 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7czl\" (UniqueName: \"kubernetes.io/projected/f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6-kube-api-access-x7czl\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:44:51.716088 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:51.716036 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6-util\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:44:51.716088 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:51.716045 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6-bundle\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:44:52.427485 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:52.427453 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l" Sep 11 14:44:52.427485 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:52.427453 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7wd4l" event={"ID":"f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6","Type":"ContainerDied","Data":"6f74ba1524cf6674adb0796832ea663e3d5f4d551701b48b0deacef48f8941bc"} Sep 11 14:44:52.427673 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:52.427506 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f74ba1524cf6674adb0796832ea663e3d5f4d551701b48b0deacef48f8941bc" Sep 11 14:44:52.642527 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:52.642494 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jncl8"] Sep 11 14:44:52.642896 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:52.642736 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6" containerName="extract" Sep 11 14:44:52.642896 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:52.642747 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6" containerName="extract" Sep 11 14:44:52.642896 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:52.642762 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6" containerName="util" Sep 11 14:44:52.642896 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:52.642767 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6" containerName="util" Sep 11 14:44:52.642896 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:52.642774 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6" containerName="pull" Sep 11 14:44:52.642896 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:52.642780 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6" containerName="pull" Sep 11 14:44:52.642896 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:52.642821 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5b0c149-2b5b-4a0d-8b69-58ca1b0df8c6" containerName="extract" Sep 11 14:44:52.646843 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:52.646826 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jncl8" Sep 11 14:44:52.649475 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:52.649452 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 11 14:44:52.649592 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:52.649530 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 11 14:44:52.649650 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:52.649623 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-w6pmk\"" Sep 11 14:44:52.655167 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:52.655145 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jncl8"] Sep 11 14:44:52.722700 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:52.722647 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e018623-bc8c-4c67-871e-8abc83bd12d9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jncl8\" (UID: \"5e018623-bc8c-4c67-871e-8abc83bd12d9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jncl8" Sep 11 14:44:52.722700 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:52.722681 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqt9s\" (UniqueName: \"kubernetes.io/projected/5e018623-bc8c-4c67-871e-8abc83bd12d9-kube-api-access-mqt9s\") pod \"cert-manager-cainjector-559cdd8c87-jncl8\" (UID: \"5e018623-bc8c-4c67-871e-8abc83bd12d9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jncl8" Sep 11 14:44:52.823941 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:52.823911 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e018623-bc8c-4c67-871e-8abc83bd12d9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jncl8\" (UID: \"5e018623-bc8c-4c67-871e-8abc83bd12d9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jncl8" Sep 11 14:44:52.824023 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:52.823953 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqt9s\" (UniqueName: \"kubernetes.io/projected/5e018623-bc8c-4c67-871e-8abc83bd12d9-kube-api-access-mqt9s\") pod \"cert-manager-cainjector-559cdd8c87-jncl8\" (UID: \"5e018623-bc8c-4c67-871e-8abc83bd12d9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jncl8" Sep 11 14:44:52.832865 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:52.832840 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e018623-bc8c-4c67-871e-8abc83bd12d9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jncl8\" (UID: \"5e018623-bc8c-4c67-871e-8abc83bd12d9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jncl8" Sep 11 14:44:52.832941 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:52.832875 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqt9s\" (UniqueName: \"kubernetes.io/projected/5e018623-bc8c-4c67-871e-8abc83bd12d9-kube-api-access-mqt9s\") pod \"cert-manager-cainjector-559cdd8c87-jncl8\" (UID: \"5e018623-bc8c-4c67-871e-8abc83bd12d9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jncl8" Sep 11 14:44:52.956170 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:52.956151 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jncl8" Sep 11 14:44:53.227084 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:53.227061 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jncl8"] Sep 11 14:44:53.229365 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:44:53.229342 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e018623_bc8c_4c67_871e_8abc83bd12d9.slice/crio-0ca93e32253d46668eaa6797cebf19511f371939f68bc95e0dcde4294369b4f9 WatchSource:0}: Error finding container 0ca93e32253d46668eaa6797cebf19511f371939f68bc95e0dcde4294369b4f9: Status 404 returned error can't find the container with id 0ca93e32253d46668eaa6797cebf19511f371939f68bc95e0dcde4294369b4f9 Sep 11 14:44:53.432330 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:53.432304 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jncl8" event={"ID":"5e018623-bc8c-4c67-871e-8abc83bd12d9","Type":"ContainerStarted","Data":"0ca93e32253d46668eaa6797cebf19511f371939f68bc95e0dcde4294369b4f9"} Sep 11 14:44:56.446430 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:56.446387 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jncl8" event={"ID":"5e018623-bc8c-4c67-871e-8abc83bd12d9","Type":"ContainerStarted","Data":"9ecda96ee636857a65eb58b7ec536d43133d0c3e84a02e5d96f6a6feb86c0225"} Sep 11 14:44:56.464342 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:56.464296 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jncl8" podStartSLOduration=1.5917915009999999 podStartE2EDuration="4.464279639s" totalImagesPullingTime="2.872488138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:44:56.463461969 +0000 UTC m=+287.707447864" watchObservedRunningTime="2026-09-11 14:44:56.464279639 +0000 UTC m=+287.708265296" Sep 11 14:44:57.444207 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:57.444172 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wcnb5"] Sep 11 14:44:57.446818 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:57.446800 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wcnb5" Sep 11 14:44:57.449209 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:57.449159 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 11 14:44:57.449329 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:57.449237 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-6tq2h\"" Sep 11 14:44:57.449994 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:57.449975 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 11 14:44:57.459696 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:57.459674 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wcnb5"] Sep 11 14:44:57.558637 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:57.558613 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxxg4\" (UniqueName: \"kubernetes.io/projected/c3014ee9-77d1-4014-a146-e79bf0630cca-kube-api-access-kxxg4\") pod \"openshift-lws-operator-fd8ccff4c-wcnb5\" (UID: \"c3014ee9-77d1-4014-a146-e79bf0630cca\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wcnb5" Sep 11 14:44:57.558760 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:57.558687 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3014ee9-77d1-4014-a146-e79bf0630cca-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wcnb5\" (UID: \"c3014ee9-77d1-4014-a146-e79bf0630cca\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wcnb5" Sep 11 14:44:57.659136 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:57.659108 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3014ee9-77d1-4014-a146-e79bf0630cca-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wcnb5\" (UID: \"c3014ee9-77d1-4014-a146-e79bf0630cca\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wcnb5" Sep 11 14:44:57.659260 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:57.659143 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxxg4\" (UniqueName: \"kubernetes.io/projected/c3014ee9-77d1-4014-a146-e79bf0630cca-kube-api-access-kxxg4\") pod \"openshift-lws-operator-fd8ccff4c-wcnb5\" (UID: \"c3014ee9-77d1-4014-a146-e79bf0630cca\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wcnb5" Sep 11 14:44:57.659469 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:57.659449 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3014ee9-77d1-4014-a146-e79bf0630cca-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wcnb5\" (UID: \"c3014ee9-77d1-4014-a146-e79bf0630cca\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wcnb5" Sep 11 14:44:57.667962 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:57.667932 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxxg4\" (UniqueName: \"kubernetes.io/projected/c3014ee9-77d1-4014-a146-e79bf0630cca-kube-api-access-kxxg4\") pod \"openshift-lws-operator-fd8ccff4c-wcnb5\" (UID: \"c3014ee9-77d1-4014-a146-e79bf0630cca\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wcnb5" Sep 11 14:44:57.755398 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:57.755336 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wcnb5" Sep 11 14:44:58.026791 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:58.026766 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wcnb5"] Sep 11 14:44:58.029020 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:44:58.028996 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3014ee9_77d1_4014_a146_e79bf0630cca.slice/crio-6210bfccd2f103d7ecc9013771201e6e6cdc2034a583945076a620d8f2f3ed61 WatchSource:0}: Error finding container 6210bfccd2f103d7ecc9013771201e6e6cdc2034a583945076a620d8f2f3ed61: Status 404 returned error can't find the container with id 6210bfccd2f103d7ecc9013771201e6e6cdc2034a583945076a620d8f2f3ed61 Sep 11 14:44:58.454977 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:44:58.454944 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wcnb5" event={"ID":"c3014ee9-77d1-4014-a146-e79bf0630cca","Type":"ContainerStarted","Data":"6210bfccd2f103d7ecc9013771201e6e6cdc2034a583945076a620d8f2f3ed61"} Sep 11 14:45:05.481595 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:05.481555 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wcnb5" event={"ID":"c3014ee9-77d1-4014-a146-e79bf0630cca","Type":"ContainerStarted","Data":"2eb1f642c9187b56409b68481a013f2c52b387204b75bf7ba941fd17885b6789"} Sep 11 14:45:05.501915 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:05.501864 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wcnb5" podStartSLOduration=1.452922485 podStartE2EDuration="8.501846967s" totalImagesPullingTime="7.048924482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:45:05.500442277 +0000 UTC m=+296.744427934" watchObservedRunningTime="2026-09-11 14:45:05.501846967 +0000 UTC m=+296.745832625" Sep 11 14:45:07.186031 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:07.185998 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl"] Sep 11 14:45:07.204581 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:07.204559 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl"] Sep 11 14:45:07.204692 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:07.204648 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl" Sep 11 14:45:07.207731 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:07.207697 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lvjqc\"" Sep 11 14:45:07.207846 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:07.207765 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 14:45:07.207846 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:07.207776 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 14:45:07.331733 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:07.331688 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtzwk\" (UniqueName: \"kubernetes.io/projected/469b5767-9460-4c50-ac47-e9fed0de88ef-kube-api-access-xtzwk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl\" (UID: \"469b5767-9460-4c50-ac47-e9fed0de88ef\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl" Sep 11 14:45:07.331835 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:07.331743 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/469b5767-9460-4c50-ac47-e9fed0de88ef-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl\" (UID: \"469b5767-9460-4c50-ac47-e9fed0de88ef\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl" Sep 11 14:45:07.331835 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:07.331765 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/469b5767-9460-4c50-ac47-e9fed0de88ef-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl\" (UID: \"469b5767-9460-4c50-ac47-e9fed0de88ef\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl" Sep 11 14:45:07.432440 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:07.432411 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtzwk\" (UniqueName: \"kubernetes.io/projected/469b5767-9460-4c50-ac47-e9fed0de88ef-kube-api-access-xtzwk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl\" (UID: \"469b5767-9460-4c50-ac47-e9fed0de88ef\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl" Sep 11 14:45:07.432560 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:07.432445 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/469b5767-9460-4c50-ac47-e9fed0de88ef-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl\" (UID: \"469b5767-9460-4c50-ac47-e9fed0de88ef\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl" Sep 11 14:45:07.432560 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:07.432466 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/469b5767-9460-4c50-ac47-e9fed0de88ef-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl\" (UID: \"469b5767-9460-4c50-ac47-e9fed0de88ef\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl" Sep 11 14:45:07.432807 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:07.432788 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/469b5767-9460-4c50-ac47-e9fed0de88ef-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl\" (UID: \"469b5767-9460-4c50-ac47-e9fed0de88ef\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl" Sep 11 14:45:07.432850 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:07.432809 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/469b5767-9460-4c50-ac47-e9fed0de88ef-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl\" (UID: \"469b5767-9460-4c50-ac47-e9fed0de88ef\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl" Sep 11 14:45:07.441623 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:07.441570 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtzwk\" (UniqueName: \"kubernetes.io/projected/469b5767-9460-4c50-ac47-e9fed0de88ef-kube-api-access-xtzwk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl\" (UID: \"469b5767-9460-4c50-ac47-e9fed0de88ef\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl" Sep 11 14:45:07.513002 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:07.512983 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl" Sep 11 14:45:07.802388 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:07.802207 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl"] Sep 11 14:45:07.804780 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:45:07.804750 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod469b5767_9460_4c50_ac47_e9fed0de88ef.slice/crio-d1f4cdc7d39d90241d501613c9c448eaf3f18bcfb71f461842bfed8631399467 WatchSource:0}: Error finding container d1f4cdc7d39d90241d501613c9c448eaf3f18bcfb71f461842bfed8631399467: Status 404 returned error can't find the container with id d1f4cdc7d39d90241d501613c9c448eaf3f18bcfb71f461842bfed8631399467 Sep 11 14:45:08.493341 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:08.493308 2583 generic.go:356] "Generic (PLEG): container finished" podID="469b5767-9460-4c50-ac47-e9fed0de88ef" containerID="e6421aba95ff16d77bcc55647099848edff7e01f2d90429e446cb289c9f0a259" exitCode=0 Sep 11 14:45:08.493656 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:08.493398 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl" event={"ID":"469b5767-9460-4c50-ac47-e9fed0de88ef","Type":"ContainerDied","Data":"e6421aba95ff16d77bcc55647099848edff7e01f2d90429e446cb289c9f0a259"} Sep 11 14:45:08.493656 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:08.493447 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl" event={"ID":"469b5767-9460-4c50-ac47-e9fed0de88ef","Type":"ContainerStarted","Data":"d1f4cdc7d39d90241d501613c9c448eaf3f18bcfb71f461842bfed8631399467"} Sep 11 14:45:09.133645 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:09.133613 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lwrt2_57d4ba26-384d-42f8-86a1-335e27cdf762/console-operator/1.log" Sep 11 14:45:09.133645 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:09.133636 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lwrt2_57d4ba26-384d-42f8-86a1-335e27cdf762/console-operator/1.log" Sep 11 14:45:09.145544 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:09.145518 2583 kubelet.go:1648] "Image garbage collection succeeded" Sep 11 14:45:09.498742 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:09.498714 2583 generic.go:356] "Generic (PLEG): container finished" podID="469b5767-9460-4c50-ac47-e9fed0de88ef" containerID="ab0bdc7109415c7ee357c4703d879d39e65eb641b70292dc260cc655147e59c4" exitCode=0 Sep 11 14:45:09.499031 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:09.498764 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl" event={"ID":"469b5767-9460-4c50-ac47-e9fed0de88ef","Type":"ContainerDied","Data":"ab0bdc7109415c7ee357c4703d879d39e65eb641b70292dc260cc655147e59c4"} Sep 11 14:45:10.504408 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:10.504372 2583 generic.go:356] "Generic (PLEG): container finished" podID="469b5767-9460-4c50-ac47-e9fed0de88ef" containerID="88ff1fb0954de112c585ba8a5f920d59ad89b814a921198f22f00eecef36b778" exitCode=0 Sep 11 14:45:10.504777 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:10.504454 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl" event={"ID":"469b5767-9460-4c50-ac47-e9fed0de88ef","Type":"ContainerDied","Data":"88ff1fb0954de112c585ba8a5f920d59ad89b814a921198f22f00eecef36b778"} Sep 11 14:45:11.627653 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:11.627629 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl" Sep 11 14:45:11.767341 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:11.767279 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtzwk\" (UniqueName: \"kubernetes.io/projected/469b5767-9460-4c50-ac47-e9fed0de88ef-kube-api-access-xtzwk\") pod \"469b5767-9460-4c50-ac47-e9fed0de88ef\" (UID: \"469b5767-9460-4c50-ac47-e9fed0de88ef\") " Sep 11 14:45:11.767341 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:11.767324 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/469b5767-9460-4c50-ac47-e9fed0de88ef-bundle\") pod \"469b5767-9460-4c50-ac47-e9fed0de88ef\" (UID: \"469b5767-9460-4c50-ac47-e9fed0de88ef\") " Sep 11 14:45:11.767341 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:11.767341 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/469b5767-9460-4c50-ac47-e9fed0de88ef-util\") pod \"469b5767-9460-4c50-ac47-e9fed0de88ef\" (UID: \"469b5767-9460-4c50-ac47-e9fed0de88ef\") " Sep 11 14:45:11.768054 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:11.768024 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469b5767-9460-4c50-ac47-e9fed0de88ef-bundle" (OuterVolumeSpecName: "bundle") pod "469b5767-9460-4c50-ac47-e9fed0de88ef" (UID: "469b5767-9460-4c50-ac47-e9fed0de88ef"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:45:11.769357 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:11.769329 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/469b5767-9460-4c50-ac47-e9fed0de88ef-kube-api-access-xtzwk" (OuterVolumeSpecName: "kube-api-access-xtzwk") pod "469b5767-9460-4c50-ac47-e9fed0de88ef" (UID: "469b5767-9460-4c50-ac47-e9fed0de88ef"). InnerVolumeSpecName "kube-api-access-xtzwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:45:11.772778 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:11.772746 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469b5767-9460-4c50-ac47-e9fed0de88ef-util" (OuterVolumeSpecName: "util") pod "469b5767-9460-4c50-ac47-e9fed0de88ef" (UID: "469b5767-9460-4c50-ac47-e9fed0de88ef"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:45:11.868210 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:11.868187 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtzwk\" (UniqueName: \"kubernetes.io/projected/469b5767-9460-4c50-ac47-e9fed0de88ef-kube-api-access-xtzwk\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:45:11.868210 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:11.868206 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/469b5767-9460-4c50-ac47-e9fed0de88ef-bundle\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:45:11.868326 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:11.868215 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/469b5767-9460-4c50-ac47-e9fed0de88ef-util\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:45:12.513631 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:12.513596 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl" Sep 11 14:45:12.513799 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:12.513603 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af86vsl" event={"ID":"469b5767-9460-4c50-ac47-e9fed0de88ef","Type":"ContainerDied","Data":"d1f4cdc7d39d90241d501613c9c448eaf3f18bcfb71f461842bfed8631399467"} Sep 11 14:45:12.513799 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:12.513736 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1f4cdc7d39d90241d501613c9c448eaf3f18bcfb71f461842bfed8631399467" Sep 11 14:45:21.304057 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:21.304025 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88"] Sep 11 14:45:21.304427 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:21.304294 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="469b5767-9460-4c50-ac47-e9fed0de88ef" containerName="util" Sep 11 14:45:21.304427 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:21.304305 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="469b5767-9460-4c50-ac47-e9fed0de88ef" containerName="util" Sep 11 14:45:21.304427 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:21.304312 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="469b5767-9460-4c50-ac47-e9fed0de88ef" containerName="pull" Sep 11 14:45:21.304427 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:21.304318 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="469b5767-9460-4c50-ac47-e9fed0de88ef" containerName="pull" Sep 11 14:45:21.304427 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:21.304332 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="469b5767-9460-4c50-ac47-e9fed0de88ef" containerName="extract" Sep 11 14:45:21.304427 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:21.304337 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="469b5767-9460-4c50-ac47-e9fed0de88ef" containerName="extract" Sep 11 14:45:21.304427 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:21.304381 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="469b5767-9460-4c50-ac47-e9fed0de88ef" containerName="extract" Sep 11 14:45:21.311647 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:21.311627 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88" Sep 11 14:45:21.314189 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:21.314164 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 14:45:21.314310 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:21.314201 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lvjqc\"" Sep 11 14:45:21.314310 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:21.314205 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 14:45:21.318201 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:21.317957 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88"] Sep 11 14:45:21.431844 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:21.431810 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88\" (UID: \"d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88" Sep 11 14:45:21.432004 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:21.431869 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ttb4\" (UniqueName: \"kubernetes.io/projected/d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce-kube-api-access-6ttb4\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88\" (UID: \"d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88" Sep 11 14:45:21.432004 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:21.431917 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88\" (UID: \"d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88" Sep 11 14:45:21.532529 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:21.532487 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ttb4\" (UniqueName: \"kubernetes.io/projected/d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce-kube-api-access-6ttb4\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88\" (UID: \"d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88" Sep 11 14:45:21.532529 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:21.532531 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88\" (UID: \"d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88" Sep 11 14:45:21.532683 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:21.532578 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88\" (UID: \"d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88" Sep 11 14:45:21.532929 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:21.532913 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88\" (UID: \"d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88" Sep 11 14:45:21.532966 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:21.532951 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88\" (UID: \"d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88" Sep 11 14:45:21.544221 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:21.544201 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ttb4\" (UniqueName: \"kubernetes.io/projected/d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce-kube-api-access-6ttb4\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88\" (UID: \"d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88" Sep 11 14:45:21.620853 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:21.620777 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88" Sep 11 14:45:21.891718 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:21.891683 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88"] Sep 11 14:45:21.894054 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:45:21.894024 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e3ad9c_a953_4f2d_9fc4_371f7236c5ce.slice/crio-ea8baa523f26a6b4a6c93ec484efcbf0b2b98e9f14f35eaca2de202003be6217 WatchSource:0}: Error finding container ea8baa523f26a6b4a6c93ec484efcbf0b2b98e9f14f35eaca2de202003be6217: Status 404 returned error can't find the container with id ea8baa523f26a6b4a6c93ec484efcbf0b2b98e9f14f35eaca2de202003be6217 Sep 11 14:45:22.558264 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:22.558232 2583 generic.go:356] "Generic (PLEG): container finished" podID="d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce" containerID="e23d976115f0de2ec743f5437715ef23f1c859fab15e5515a663ebb1b66baf55" exitCode=0 Sep 11 14:45:22.558572 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:22.558312 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88" event={"ID":"d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce","Type":"ContainerDied","Data":"e23d976115f0de2ec743f5437715ef23f1c859fab15e5515a663ebb1b66baf55"} Sep 11 14:45:22.558572 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:22.558361 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88" event={"ID":"d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce","Type":"ContainerStarted","Data":"ea8baa523f26a6b4a6c93ec484efcbf0b2b98e9f14f35eaca2de202003be6217"} Sep 11 14:45:22.558726 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:22.558698 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:45:23.564254 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:23.564224 2583 generic.go:356] "Generic (PLEG): container finished" podID="d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce" containerID="fc0a9e83275607c9f24fe42881dae1f00a3e9654e77f5386730b1c91ae8943b8" exitCode=0 Sep 11 14:45:23.564595 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:23.564307 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88" event={"ID":"d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce","Type":"ContainerDied","Data":"fc0a9e83275607c9f24fe42881dae1f00a3e9654e77f5386730b1c91ae8943b8"} Sep 11 14:45:23.918457 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:23.918419 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67f987f856-vt2v2"] Sep 11 14:45:23.921487 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:23.921470 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-vt2v2" Sep 11 14:45:23.924876 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:23.924848 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zjcsv\"" Sep 11 14:45:23.925031 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:23.925011 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 11 14:45:23.925087 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:23.925027 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 11 14:45:23.925138 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:23.925035 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 11 14:45:23.925138 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:23.925100 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 11 14:45:23.956033 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:23.956005 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67f987f856-vt2v2"] Sep 11 14:45:24.051625 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:24.051590 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63e3a376-f5d7-44f7-b725-e2f4ec532edc-webhook-cert\") pod \"opendatahub-operator-controller-manager-67f987f856-vt2v2\" (UID: \"63e3a376-f5d7-44f7-b725-e2f4ec532edc\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-vt2v2" Sep 11 14:45:24.051779 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:24.051635 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63e3a376-f5d7-44f7-b725-e2f4ec532edc-tmp\") pod \"opendatahub-operator-controller-manager-67f987f856-vt2v2\" (UID: \"63e3a376-f5d7-44f7-b725-e2f4ec532edc\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-vt2v2" Sep 11 14:45:24.051779 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:24.051654 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/63e3a376-f5d7-44f7-b725-e2f4ec532edc-manifests\") pod \"opendatahub-operator-controller-manager-67f987f856-vt2v2\" (UID: \"63e3a376-f5d7-44f7-b725-e2f4ec532edc\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-vt2v2" Sep 11 14:45:24.051779 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:24.051756 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk4nl\" (UniqueName: \"kubernetes.io/projected/63e3a376-f5d7-44f7-b725-e2f4ec532edc-kube-api-access-bk4nl\") pod \"opendatahub-operator-controller-manager-67f987f856-vt2v2\" (UID: \"63e3a376-f5d7-44f7-b725-e2f4ec532edc\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-vt2v2" Sep 11 14:45:24.051925 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:24.051838 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/63e3a376-f5d7-44f7-b725-e2f4ec532edc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67f987f856-vt2v2\" (UID: \"63e3a376-f5d7-44f7-b725-e2f4ec532edc\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-vt2v2" Sep 11 14:45:24.152879 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:24.152854 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/63e3a376-f5d7-44f7-b725-e2f4ec532edc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67f987f856-vt2v2\" (UID: \"63e3a376-f5d7-44f7-b725-e2f4ec532edc\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-vt2v2" Sep 11 14:45:24.152997 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:24.152904 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63e3a376-f5d7-44f7-b725-e2f4ec532edc-webhook-cert\") pod \"opendatahub-operator-controller-manager-67f987f856-vt2v2\" (UID: \"63e3a376-f5d7-44f7-b725-e2f4ec532edc\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-vt2v2" Sep 11 14:45:24.152997 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:24.152932 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63e3a376-f5d7-44f7-b725-e2f4ec532edc-tmp\") pod \"opendatahub-operator-controller-manager-67f987f856-vt2v2\" (UID: \"63e3a376-f5d7-44f7-b725-e2f4ec532edc\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-vt2v2" Sep 11 14:45:24.153097 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:24.153024 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/63e3a376-f5d7-44f7-b725-e2f4ec532edc-manifests\") pod \"opendatahub-operator-controller-manager-67f987f856-vt2v2\" (UID: \"63e3a376-f5d7-44f7-b725-e2f4ec532edc\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-vt2v2" Sep 11 14:45:24.153097 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:24.153065 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk4nl\" (UniqueName: \"kubernetes.io/projected/63e3a376-f5d7-44f7-b725-e2f4ec532edc-kube-api-access-bk4nl\") pod \"opendatahub-operator-controller-manager-67f987f856-vt2v2\" (UID: \"63e3a376-f5d7-44f7-b725-e2f4ec532edc\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-vt2v2" Sep 11 14:45:24.153390 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:24.153369 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/63e3a376-f5d7-44f7-b725-e2f4ec532edc-manifests\") pod \"opendatahub-operator-controller-manager-67f987f856-vt2v2\" (UID: \"63e3a376-f5d7-44f7-b725-e2f4ec532edc\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-vt2v2" Sep 11 14:45:24.155174 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:24.155158 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63e3a376-f5d7-44f7-b725-e2f4ec532edc-tmp\") pod \"opendatahub-operator-controller-manager-67f987f856-vt2v2\" (UID: \"63e3a376-f5d7-44f7-b725-e2f4ec532edc\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-vt2v2" Sep 11 14:45:24.155375 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:24.155356 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63e3a376-f5d7-44f7-b725-e2f4ec532edc-webhook-cert\") pod \"opendatahub-operator-controller-manager-67f987f856-vt2v2\" (UID: \"63e3a376-f5d7-44f7-b725-e2f4ec532edc\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-vt2v2" Sep 11 14:45:24.155412 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:24.155356 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/63e3a376-f5d7-44f7-b725-e2f4ec532edc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67f987f856-vt2v2\" (UID: \"63e3a376-f5d7-44f7-b725-e2f4ec532edc\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-vt2v2" Sep 11 14:45:24.164018 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:24.163993 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk4nl\" (UniqueName: \"kubernetes.io/projected/63e3a376-f5d7-44f7-b725-e2f4ec532edc-kube-api-access-bk4nl\") pod \"opendatahub-operator-controller-manager-67f987f856-vt2v2\" (UID: \"63e3a376-f5d7-44f7-b725-e2f4ec532edc\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-vt2v2" Sep 11 14:45:24.231253 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:24.231205 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-vt2v2" Sep 11 14:45:24.509420 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:24.509352 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67f987f856-vt2v2"] Sep 11 14:45:24.512251 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:45:24.512222 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e3a376_f5d7_44f7_b725_e2f4ec532edc.slice/crio-40b134340a82fd21b6267a6175123764eb8a3fa568caf20dfc51b1170c52ccdd WatchSource:0}: Error finding container 40b134340a82fd21b6267a6175123764eb8a3fa568caf20dfc51b1170c52ccdd: Status 404 returned error can't find the container with id 40b134340a82fd21b6267a6175123764eb8a3fa568caf20dfc51b1170c52ccdd Sep 11 14:45:24.569892 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:24.569865 2583 generic.go:356] "Generic (PLEG): container finished" podID="d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce" containerID="d5bf34c52749206a5100b7413dc2f0dc75b66dd3a97e31102f7aabb5d1b2e01c" exitCode=0 Sep 11 14:45:24.570212 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:24.569941 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88" event={"ID":"d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce","Type":"ContainerDied","Data":"d5bf34c52749206a5100b7413dc2f0dc75b66dd3a97e31102f7aabb5d1b2e01c"} Sep 11 14:45:24.571225 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:24.571202 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-vt2v2" event={"ID":"63e3a376-f5d7-44f7-b725-e2f4ec532edc","Type":"ContainerStarted","Data":"40b134340a82fd21b6267a6175123764eb8a3fa568caf20dfc51b1170c52ccdd"} Sep 11 14:45:25.729808 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:25.729776 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88" Sep 11 14:45:25.866595 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:25.866524 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce-util\") pod \"d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce\" (UID: \"d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce\") " Sep 11 14:45:25.866595 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:25.866584 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ttb4\" (UniqueName: \"kubernetes.io/projected/d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce-kube-api-access-6ttb4\") pod \"d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce\" (UID: \"d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce\") " Sep 11 14:45:25.866790 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:25.866636 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce-bundle\") pod \"d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce\" (UID: \"d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce\") " Sep 11 14:45:25.867732 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:25.867676 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce-bundle" (OuterVolumeSpecName: "bundle") pod "d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce" (UID: "d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:45:25.870056 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:25.870013 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce-kube-api-access-6ttb4" (OuterVolumeSpecName: "kube-api-access-6ttb4") pod "d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce" (UID: "d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce"). InnerVolumeSpecName "kube-api-access-6ttb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:45:25.874050 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:25.874026 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce-util" (OuterVolumeSpecName: "util") pod "d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce" (UID: "d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:45:25.967937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:25.967900 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce-bundle\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:45:25.967937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:25.967929 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce-util\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:45:25.967937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:25.967939 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ttb4\" (UniqueName: \"kubernetes.io/projected/d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce-kube-api-access-6ttb4\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:45:26.583218 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:26.583179 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88" event={"ID":"d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce","Type":"ContainerDied","Data":"ea8baa523f26a6b4a6c93ec484efcbf0b2b98e9f14f35eaca2de202003be6217"} Sep 11 14:45:26.583218 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:26.583206 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n7m88" Sep 11 14:45:26.583429 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:26.583236 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea8baa523f26a6b4a6c93ec484efcbf0b2b98e9f14f35eaca2de202003be6217" Sep 11 14:45:27.589033 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:27.588995 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-vt2v2" event={"ID":"63e3a376-f5d7-44f7-b725-e2f4ec532edc","Type":"ContainerStarted","Data":"8e7f2fdee90dab557c8e1c8eff29116d0a9fed0e4042c33c2afcfe86f952d420"} Sep 11 14:45:28.594826 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:28.594732 2583 generic.go:356] "Generic (PLEG): container finished" podID="63e3a376-f5d7-44f7-b725-e2f4ec532edc" containerID="8e7f2fdee90dab557c8e1c8eff29116d0a9fed0e4042c33c2afcfe86f952d420" exitCode=0 Sep 11 14:45:28.594826 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:28.594806 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-vt2v2" event={"ID":"63e3a376-f5d7-44f7-b725-e2f4ec532edc","Type":"ContainerDied","Data":"8e7f2fdee90dab557c8e1c8eff29116d0a9fed0e4042c33c2afcfe86f952d420"} Sep 11 14:45:29.601035 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:29.600996 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-vt2v2" event={"ID":"63e3a376-f5d7-44f7-b725-e2f4ec532edc","Type":"ContainerStarted","Data":"4d33ef8723684892f1ad1d23ebfe87c837e8c6f7027297a3dae029f907fb1431"} Sep 11 14:45:29.601438 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:29.601104 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-vt2v2" Sep 11 14:45:29.655099 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:29.655048 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-vt2v2" podStartSLOduration=2.569688451 podStartE2EDuration="6.655028396s" totalImagesPullingTime="3.171242532s" totalInitContainerRuntime="914.097413ms" isStatefulPod=false podCreationTimestamp="2026-09-11 14:45:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:45:29.651519111 +0000 UTC m=+320.895504768" watchObservedRunningTime="2026-09-11 14:45:29.655028396 +0000 UTC m=+320.899014056" Sep 11 14:45:40.607000 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:40.606967 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-vt2v2" Sep 11 14:45:42.630747 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:42.630699 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c79f8858d-vxvs5"] Sep 11 14:45:42.631182 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:42.631080 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce" containerName="pull" Sep 11 14:45:42.631182 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:42.631097 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce" containerName="pull" Sep 11 14:45:42.631182 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:42.631122 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce" containerName="extract" Sep 11 14:45:42.631182 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:42.631131 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce" containerName="extract" Sep 11 14:45:42.631182 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:42.631142 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce" containerName="util" Sep 11 14:45:42.631182 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:42.631149 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce" containerName="util" Sep 11 14:45:42.631463 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:42.631211 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9e3ad9c-a953-4f2d-9fc4-371f7236c5ce" containerName="extract" Sep 11 14:45:42.638265 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:42.638240 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-vxvs5" Sep 11 14:45:42.648057 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:42.648039 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 11 14:45:42.648298 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:42.648280 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 11 14:45:42.648951 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:42.648936 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-j64lt\"" Sep 11 14:45:42.649312 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:42.649290 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 11 14:45:42.651002 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:42.650982 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c79f8858d-vxvs5"] Sep 11 14:45:42.784813 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:42.784774 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2dvd\" (UniqueName: \"kubernetes.io/projected/10d337bb-1bc5-425f-8124-ab8afed6e1cc-kube-api-access-f2dvd\") pod \"lws-controller-manager-6c79f8858d-vxvs5\" (UID: \"10d337bb-1bc5-425f-8124-ab8afed6e1cc\") " pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-vxvs5" Sep 11 14:45:42.784998 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:42.784854 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10d337bb-1bc5-425f-8124-ab8afed6e1cc-cert\") pod \"lws-controller-manager-6c79f8858d-vxvs5\" (UID: \"10d337bb-1bc5-425f-8124-ab8afed6e1cc\") " pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-vxvs5" Sep 11 14:45:42.784998 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:42.784901 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/10d337bb-1bc5-425f-8124-ab8afed6e1cc-manager-config\") pod \"lws-controller-manager-6c79f8858d-vxvs5\" (UID: \"10d337bb-1bc5-425f-8124-ab8afed6e1cc\") " pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-vxvs5" Sep 11 14:45:42.784998 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:42.784927 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/10d337bb-1bc5-425f-8124-ab8afed6e1cc-metrics-cert\") pod \"lws-controller-manager-6c79f8858d-vxvs5\" (UID: \"10d337bb-1bc5-425f-8124-ab8afed6e1cc\") " pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-vxvs5" Sep 11 14:45:42.885427 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:42.885352 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/10d337bb-1bc5-425f-8124-ab8afed6e1cc-metrics-cert\") pod \"lws-controller-manager-6c79f8858d-vxvs5\" (UID: \"10d337bb-1bc5-425f-8124-ab8afed6e1cc\") " pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-vxvs5" Sep 11 14:45:42.885427 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:42.885401 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2dvd\" (UniqueName: \"kubernetes.io/projected/10d337bb-1bc5-425f-8124-ab8afed6e1cc-kube-api-access-f2dvd\") pod \"lws-controller-manager-6c79f8858d-vxvs5\" (UID: \"10d337bb-1bc5-425f-8124-ab8afed6e1cc\") " pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-vxvs5" Sep 11 14:45:42.885620 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:42.885433 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10d337bb-1bc5-425f-8124-ab8afed6e1cc-cert\") pod \"lws-controller-manager-6c79f8858d-vxvs5\" (UID: \"10d337bb-1bc5-425f-8124-ab8afed6e1cc\") " pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-vxvs5" Sep 11 14:45:42.885620 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:42.885456 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/10d337bb-1bc5-425f-8124-ab8afed6e1cc-manager-config\") pod \"lws-controller-manager-6c79f8858d-vxvs5\" (UID: \"10d337bb-1bc5-425f-8124-ab8afed6e1cc\") " pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-vxvs5" Sep 11 14:45:42.886088 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:42.886067 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/10d337bb-1bc5-425f-8124-ab8afed6e1cc-manager-config\") pod \"lws-controller-manager-6c79f8858d-vxvs5\" (UID: \"10d337bb-1bc5-425f-8124-ab8afed6e1cc\") " pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-vxvs5" Sep 11 14:45:42.887874 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:42.887853 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/10d337bb-1bc5-425f-8124-ab8afed6e1cc-metrics-cert\") pod \"lws-controller-manager-6c79f8858d-vxvs5\" (UID: \"10d337bb-1bc5-425f-8124-ab8afed6e1cc\") " pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-vxvs5" Sep 11 14:45:42.887955 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:42.887878 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10d337bb-1bc5-425f-8124-ab8afed6e1cc-cert\") pod \"lws-controller-manager-6c79f8858d-vxvs5\" (UID: \"10d337bb-1bc5-425f-8124-ab8afed6e1cc\") " pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-vxvs5" Sep 11 14:45:42.895336 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:42.895316 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2dvd\" (UniqueName: \"kubernetes.io/projected/10d337bb-1bc5-425f-8124-ab8afed6e1cc-kube-api-access-f2dvd\") pod \"lws-controller-manager-6c79f8858d-vxvs5\" (UID: \"10d337bb-1bc5-425f-8124-ab8afed6e1cc\") " pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-vxvs5" Sep 11 14:45:42.947265 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:42.947240 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-vxvs5" Sep 11 14:45:43.223123 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:43.222961 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c79f8858d-vxvs5"] Sep 11 14:45:43.225122 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:45:43.225092 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10d337bb_1bc5_425f_8124_ab8afed6e1cc.slice/crio-321d584fac01665cb0c875e4ff573b749e0e23627519d46b2e91f5a4ab919913 WatchSource:0}: Error finding container 321d584fac01665cb0c875e4ff573b749e0e23627519d46b2e91f5a4ab919913: Status 404 returned error can't find the container with id 321d584fac01665cb0c875e4ff573b749e0e23627519d46b2e91f5a4ab919913 Sep 11 14:45:43.667833 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:43.667801 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-vxvs5" event={"ID":"10d337bb-1bc5-425f-8124-ab8afed6e1cc","Type":"ContainerStarted","Data":"321d584fac01665cb0c875e4ff573b749e0e23627519d46b2e91f5a4ab919913"} Sep 11 14:45:45.679007 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:45.678969 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-vxvs5" event={"ID":"10d337bb-1bc5-425f-8124-ab8afed6e1cc","Type":"ContainerStarted","Data":"9ff4760351ae9a4069ecd5f0282cbe09365fd40f1219446ff861c3669310eade"} Sep 11 14:45:45.679408 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:45.679107 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-vxvs5" Sep 11 14:45:51.910343 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:51.910293 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-vxvs5" podStartSLOduration=8.372813615 podStartE2EDuration="9.910277251s" totalImagesPullingTime="1.537463636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:45:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:45:45.715421752 +0000 UTC m=+336.959407409" watchObservedRunningTime="2026-09-11 14:45:51.910277251 +0000 UTC m=+343.154262916" Sep 11 14:45:51.910882 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:51.910862 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h"] Sep 11 14:45:51.914056 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:51.914040 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:51.916789 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:51.916764 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-kpftd\"" Sep 11 14:45:51.916925 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:51.916904 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 11 14:45:51.917478 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:51.917463 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 11 14:45:51.917565 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:51.917465 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 14:45:51.928750 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:51.928731 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h"] Sep 11 14:45:51.953945 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:51.953922 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe579da6-e7a4-4a75-aea1-c2957e757fa9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h\" (UID: \"fe579da6-e7a4-4a75-aea1-c2957e757fa9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:51.954030 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:51.953965 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe579da6-e7a4-4a75-aea1-c2957e757fa9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h\" (UID: \"fe579da6-e7a4-4a75-aea1-c2957e757fa9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:51.954030 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:51.953987 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe579da6-e7a4-4a75-aea1-c2957e757fa9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h\" (UID: \"fe579da6-e7a4-4a75-aea1-c2957e757fa9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:51.954030 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:51.954006 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe579da6-e7a4-4a75-aea1-c2957e757fa9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h\" (UID: \"fe579da6-e7a4-4a75-aea1-c2957e757fa9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:51.954142 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:51.954038 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvmhc\" (UniqueName: \"kubernetes.io/projected/fe579da6-e7a4-4a75-aea1-c2957e757fa9-kube-api-access-fvmhc\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h\" (UID: \"fe579da6-e7a4-4a75-aea1-c2957e757fa9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:51.954142 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:51.954116 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe579da6-e7a4-4a75-aea1-c2957e757fa9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h\" (UID: \"fe579da6-e7a4-4a75-aea1-c2957e757fa9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:51.954206 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:51.954150 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe579da6-e7a4-4a75-aea1-c2957e757fa9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h\" (UID: \"fe579da6-e7a4-4a75-aea1-c2957e757fa9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:51.954206 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:51.954167 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe579da6-e7a4-4a75-aea1-c2957e757fa9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h\" (UID: \"fe579da6-e7a4-4a75-aea1-c2957e757fa9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:51.954269 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:51.954220 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe579da6-e7a4-4a75-aea1-c2957e757fa9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h\" (UID: \"fe579da6-e7a4-4a75-aea1-c2957e757fa9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:52.054650 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:52.054623 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe579da6-e7a4-4a75-aea1-c2957e757fa9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h\" (UID: \"fe579da6-e7a4-4a75-aea1-c2957e757fa9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:52.054774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:52.054653 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe579da6-e7a4-4a75-aea1-c2957e757fa9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h\" (UID: \"fe579da6-e7a4-4a75-aea1-c2957e757fa9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:52.054774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:52.054669 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvmhc\" (UniqueName: \"kubernetes.io/projected/fe579da6-e7a4-4a75-aea1-c2957e757fa9-kube-api-access-fvmhc\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h\" (UID: \"fe579da6-e7a4-4a75-aea1-c2957e757fa9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:52.054774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:52.054686 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe579da6-e7a4-4a75-aea1-c2957e757fa9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h\" (UID: \"fe579da6-e7a4-4a75-aea1-c2957e757fa9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:52.054774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:52.054729 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe579da6-e7a4-4a75-aea1-c2957e757fa9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h\" (UID: \"fe579da6-e7a4-4a75-aea1-c2957e757fa9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:52.054774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:52.054750 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe579da6-e7a4-4a75-aea1-c2957e757fa9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h\" (UID: \"fe579da6-e7a4-4a75-aea1-c2957e757fa9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:52.054774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:52.054776 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe579da6-e7a4-4a75-aea1-c2957e757fa9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h\" (UID: \"fe579da6-e7a4-4a75-aea1-c2957e757fa9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:52.055057 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:52.054813 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe579da6-e7a4-4a75-aea1-c2957e757fa9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h\" (UID: \"fe579da6-e7a4-4a75-aea1-c2957e757fa9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:52.055162 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:52.055073 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe579da6-e7a4-4a75-aea1-c2957e757fa9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h\" (UID: \"fe579da6-e7a4-4a75-aea1-c2957e757fa9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:52.055162 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:52.055072 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe579da6-e7a4-4a75-aea1-c2957e757fa9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h\" (UID: \"fe579da6-e7a4-4a75-aea1-c2957e757fa9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:52.055269 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:52.055161 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe579da6-e7a4-4a75-aea1-c2957e757fa9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h\" (UID: \"fe579da6-e7a4-4a75-aea1-c2957e757fa9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:52.055323 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:52.055276 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe579da6-e7a4-4a75-aea1-c2957e757fa9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h\" (UID: \"fe579da6-e7a4-4a75-aea1-c2957e757fa9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:52.055323 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:52.055307 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe579da6-e7a4-4a75-aea1-c2957e757fa9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h\" (UID: \"fe579da6-e7a4-4a75-aea1-c2957e757fa9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:52.055567 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:52.055545 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe579da6-e7a4-4a75-aea1-c2957e757fa9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h\" (UID: \"fe579da6-e7a4-4a75-aea1-c2957e757fa9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:52.056834 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:52.056813 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe579da6-e7a4-4a75-aea1-c2957e757fa9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h\" (UID: \"fe579da6-e7a4-4a75-aea1-c2957e757fa9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:52.057088 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:52.057071 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe579da6-e7a4-4a75-aea1-c2957e757fa9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h\" (UID: \"fe579da6-e7a4-4a75-aea1-c2957e757fa9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:52.066559 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:52.066534 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe579da6-e7a4-4a75-aea1-c2957e757fa9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h\" (UID: \"fe579da6-e7a4-4a75-aea1-c2957e757fa9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:52.066803 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:52.066785 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvmhc\" (UniqueName: \"kubernetes.io/projected/fe579da6-e7a4-4a75-aea1-c2957e757fa9-kube-api-access-fvmhc\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h\" (UID: \"fe579da6-e7a4-4a75-aea1-c2957e757fa9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:52.224519 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:52.224461 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:52.502800 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:52.502718 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h"] Sep 11 14:45:52.506485 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:45:52.506458 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe579da6_e7a4_4a75_aea1_c2957e757fa9.slice/crio-6fca02633ba48477fdf17889e06c41fae47867ad008b76fdfa5a2bff47ba9b34 WatchSource:0}: Error finding container 6fca02633ba48477fdf17889e06c41fae47867ad008b76fdfa5a2bff47ba9b34: Status 404 returned error can't find the container with id 6fca02633ba48477fdf17889e06c41fae47867ad008b76fdfa5a2bff47ba9b34 Sep 11 14:45:52.709565 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:52.709535 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" event={"ID":"fe579da6-e7a4-4a75-aea1-c2957e757fa9","Type":"ContainerStarted","Data":"6fca02633ba48477fdf17889e06c41fae47867ad008b76fdfa5a2bff47ba9b34"} Sep 11 14:45:54.687219 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:54.687183 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-r846f"] Sep 11 14:45:54.694207 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:54.694180 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-r846f" Sep 11 14:45:54.696729 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:54.696691 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 11 14:45:54.696729 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:54.696722 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-lhl6b\"" Sep 11 14:45:54.703072 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:54.702894 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-r846f"] Sep 11 14:45:54.777081 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:54.777042 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfgqj\" (UniqueName: \"kubernetes.io/projected/ab41be90-a53b-4678-8c6b-9f3f8ea30dbf-kube-api-access-jfgqj\") pod \"odh-model-controller-7998c67dc4-r846f\" (UID: \"ab41be90-a53b-4678-8c6b-9f3f8ea30dbf\") " pod="opendatahub/odh-model-controller-7998c67dc4-r846f" Sep 11 14:45:54.777260 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:54.777123 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab41be90-a53b-4678-8c6b-9f3f8ea30dbf-cert\") pod \"odh-model-controller-7998c67dc4-r846f\" (UID: \"ab41be90-a53b-4678-8c6b-9f3f8ea30dbf\") " pod="opendatahub/odh-model-controller-7998c67dc4-r846f" Sep 11 14:45:54.878346 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:54.878307 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfgqj\" (UniqueName: \"kubernetes.io/projected/ab41be90-a53b-4678-8c6b-9f3f8ea30dbf-kube-api-access-jfgqj\") pod \"odh-model-controller-7998c67dc4-r846f\" (UID: \"ab41be90-a53b-4678-8c6b-9f3f8ea30dbf\") " pod="opendatahub/odh-model-controller-7998c67dc4-r846f" Sep 11 14:45:54.878528 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:54.878364 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab41be90-a53b-4678-8c6b-9f3f8ea30dbf-cert\") pod \"odh-model-controller-7998c67dc4-r846f\" (UID: \"ab41be90-a53b-4678-8c6b-9f3f8ea30dbf\") " pod="opendatahub/odh-model-controller-7998c67dc4-r846f" Sep 11 14:45:54.878528 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:45:54.878491 2583 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 11 14:45:54.878637 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:45:54.878557 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab41be90-a53b-4678-8c6b-9f3f8ea30dbf-cert podName:ab41be90-a53b-4678-8c6b-9f3f8ea30dbf nodeName:}" failed. No retries permitted until 2026-09-11 14:45:55.378534737 +0000 UTC m=+346.622520393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ab41be90-a53b-4678-8c6b-9f3f8ea30dbf-cert") pod "odh-model-controller-7998c67dc4-r846f" (UID: "ab41be90-a53b-4678-8c6b-9f3f8ea30dbf") : secret "odh-model-controller-webhook-cert" not found Sep 11 14:45:54.889606 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:54.889567 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfgqj\" (UniqueName: \"kubernetes.io/projected/ab41be90-a53b-4678-8c6b-9f3f8ea30dbf-kube-api-access-jfgqj\") pod \"odh-model-controller-7998c67dc4-r846f\" (UID: \"ab41be90-a53b-4678-8c6b-9f3f8ea30dbf\") " pod="opendatahub/odh-model-controller-7998c67dc4-r846f" Sep 11 14:45:55.383941 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:55.383901 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab41be90-a53b-4678-8c6b-9f3f8ea30dbf-cert\") pod \"odh-model-controller-7998c67dc4-r846f\" (UID: \"ab41be90-a53b-4678-8c6b-9f3f8ea30dbf\") " pod="opendatahub/odh-model-controller-7998c67dc4-r846f" Sep 11 14:45:55.390692 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:55.386531 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab41be90-a53b-4678-8c6b-9f3f8ea30dbf-cert\") pod \"odh-model-controller-7998c67dc4-r846f\" (UID: \"ab41be90-a53b-4678-8c6b-9f3f8ea30dbf\") " pod="opendatahub/odh-model-controller-7998c67dc4-r846f" Sep 11 14:45:55.407214 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:55.407180 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-s7722"] Sep 11 14:45:55.409966 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:55.409947 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-s7722" Sep 11 14:45:55.413046 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:55.413024 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 11 14:45:55.422416 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:55.422392 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-s7722"] Sep 11 14:45:55.441904 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:55.441867 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:45:55.441995 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:55.441945 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:45:55.441995 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:55.441984 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:45:55.484391 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:55.484364 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9t26\" (UniqueName: \"kubernetes.io/projected/ddc27eeb-1fbb-4110-a45d-c255ed11282c-kube-api-access-t9t26\") pod \"model-serving-api-54d89bd5fc-s7722\" (UID: \"ddc27eeb-1fbb-4110-a45d-c255ed11282c\") " pod="opendatahub/model-serving-api-54d89bd5fc-s7722" Sep 11 14:45:55.484756 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:55.484688 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddc27eeb-1fbb-4110-a45d-c255ed11282c-tls-certs\") pod \"model-serving-api-54d89bd5fc-s7722\" (UID: \"ddc27eeb-1fbb-4110-a45d-c255ed11282c\") " pod="opendatahub/model-serving-api-54d89bd5fc-s7722" Sep 11 14:45:55.585676 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:55.585642 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddc27eeb-1fbb-4110-a45d-c255ed11282c-tls-certs\") pod \"model-serving-api-54d89bd5fc-s7722\" (UID: \"ddc27eeb-1fbb-4110-a45d-c255ed11282c\") " pod="opendatahub/model-serving-api-54d89bd5fc-s7722" Sep 11 14:45:55.585868 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:55.585741 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9t26\" (UniqueName: \"kubernetes.io/projected/ddc27eeb-1fbb-4110-a45d-c255ed11282c-kube-api-access-t9t26\") pod \"model-serving-api-54d89bd5fc-s7722\" (UID: \"ddc27eeb-1fbb-4110-a45d-c255ed11282c\") " pod="opendatahub/model-serving-api-54d89bd5fc-s7722" Sep 11 14:45:55.588064 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:55.588040 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddc27eeb-1fbb-4110-a45d-c255ed11282c-tls-certs\") pod \"model-serving-api-54d89bd5fc-s7722\" (UID: \"ddc27eeb-1fbb-4110-a45d-c255ed11282c\") " pod="opendatahub/model-serving-api-54d89bd5fc-s7722" Sep 11 14:45:55.594541 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:55.594514 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9t26\" (UniqueName: \"kubernetes.io/projected/ddc27eeb-1fbb-4110-a45d-c255ed11282c-kube-api-access-t9t26\") pod \"model-serving-api-54d89bd5fc-s7722\" (UID: \"ddc27eeb-1fbb-4110-a45d-c255ed11282c\") " pod="opendatahub/model-serving-api-54d89bd5fc-s7722" Sep 11 14:45:55.607294 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:55.607272 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-r846f" Sep 11 14:45:55.719487 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:55.719457 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-s7722" Sep 11 14:45:55.726474 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:55.725726 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" event={"ID":"fe579da6-e7a4-4a75-aea1-c2957e757fa9","Type":"ContainerStarted","Data":"8f02eca0770d1344ff3911c941636e0c0f24f934980b41276b925441b2308a40"} Sep 11 14:45:55.892331 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:55.892270 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" podStartSLOduration=1.95908931 podStartE2EDuration="4.892248333s" totalImagesPullingTime="2.933159023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:45:55.75069725 +0000 UTC m=+346.994682920" watchObservedRunningTime="2026-09-11 14:45:55.892248333 +0000 UTC m=+347.136233995" Sep 11 14:45:55.894523 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:55.894493 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-r846f"] Sep 11 14:45:55.897346 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:45:55.897320 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab41be90_a53b_4678_8c6b_9f3f8ea30dbf.slice/crio-2a2214290ba39942aefd7b7275f8d44a6f7158f4776da480f51d5e1d62e8801b WatchSource:0}: Error finding container 2a2214290ba39942aefd7b7275f8d44a6f7158f4776da480f51d5e1d62e8801b: Status 404 returned error can't find the container with id 2a2214290ba39942aefd7b7275f8d44a6f7158f4776da480f51d5e1d62e8801b Sep 11 14:45:55.997384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:55.997362 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-s7722"] Sep 11 14:45:55.999448 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:45:55.999421 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddc27eeb_1fbb_4110_a45d_c255ed11282c.slice/crio-48c794b84343d23d6852212f6bfe30f83d75f095a25fd014a6f26058c726ca22 WatchSource:0}: Error finding container 48c794b84343d23d6852212f6bfe30f83d75f095a25fd014a6f26058c726ca22: Status 404 returned error can't find the container with id 48c794b84343d23d6852212f6bfe30f83d75f095a25fd014a6f26058c726ca22 Sep 11 14:45:56.225391 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:56.225353 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:56.230058 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:56.230022 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:56.685627 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:56.685587 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6c79f8858d-vxvs5" Sep 11 14:45:56.740148 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:56.740009 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-r846f" event={"ID":"ab41be90-a53b-4678-8c6b-9f3f8ea30dbf","Type":"ContainerStarted","Data":"2a2214290ba39942aefd7b7275f8d44a6f7158f4776da480f51d5e1d62e8801b"} Sep 11 14:45:56.747650 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:56.747424 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-s7722" event={"ID":"ddc27eeb-1fbb-4110-a45d-c255ed11282c","Type":"ContainerStarted","Data":"48c794b84343d23d6852212f6bfe30f83d75f095a25fd014a6f26058c726ca22"} Sep 11 14:45:56.747650 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:56.747486 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:45:56.749981 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:45:56.749743 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h" Sep 11 14:46:00.772791 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:46:00.772749 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-s7722" event={"ID":"ddc27eeb-1fbb-4110-a45d-c255ed11282c","Type":"ContainerStarted","Data":"17780706362e69d5f5bc4706a20a2067da800d718972d5ab4405c95f04910b69"} Sep 11 14:46:00.773226 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:46:00.772860 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-s7722" Sep 11 14:46:00.774301 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:46:00.774272 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-r846f" event={"ID":"ab41be90-a53b-4678-8c6b-9f3f8ea30dbf","Type":"ContainerStarted","Data":"123522424a95c566a8bf0dc2501b0e7b3511a8126eb4ce55ac031b109180238f"} Sep 11 14:46:00.774403 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:46:00.774383 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-r846f" Sep 11 14:46:00.792769 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:46:00.792721 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-s7722" podStartSLOduration=1.7550346669999999 podStartE2EDuration="5.792695757s" totalImagesPullingTime="4.03766109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:45:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:46:00.790629426 +0000 UTC m=+352.034615086" watchObservedRunningTime="2026-09-11 14:46:00.792695757 +0000 UTC m=+352.036681415" Sep 11 14:46:00.809837 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:46:00.809795 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-r846f" podStartSLOduration=2.70842753 podStartE2EDuration="6.809782064s" totalImagesPullingTime="4.101354534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:45:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:46:00.808742842 +0000 UTC m=+352.052728499" watchObservedRunningTime="2026-09-11 14:46:00.809782064 +0000 UTC m=+352.053767722" Sep 11 14:46:05.361420 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:46:05.361342 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-q9lp8"] Sep 11 14:46:05.364174 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:46:05.364153 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q9lp8" Sep 11 14:46:05.367815 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:46:05.367790 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-tls47\"" Sep 11 14:46:05.367923 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:46:05.367790 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 11 14:46:05.380067 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:46:05.380046 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-q9lp8"] Sep 11 14:46:05.474373 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:46:05.474333 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9bb7c91-9123-4dc4-a469-7b7af81d9d4d-cert\") pod \"kserve-controller-manager-d6f4b65b8-q9lp8\" (UID: \"a9bb7c91-9123-4dc4-a469-7b7af81d9d4d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q9lp8" Sep 11 14:46:05.474662 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:46:05.474642 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l69kd\" (UniqueName: \"kubernetes.io/projected/a9bb7c91-9123-4dc4-a469-7b7af81d9d4d-kube-api-access-l69kd\") pod \"kserve-controller-manager-d6f4b65b8-q9lp8\" (UID: \"a9bb7c91-9123-4dc4-a469-7b7af81d9d4d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q9lp8" Sep 11 14:46:05.575104 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:46:05.575075 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l69kd\" (UniqueName: \"kubernetes.io/projected/a9bb7c91-9123-4dc4-a469-7b7af81d9d4d-kube-api-access-l69kd\") pod \"kserve-controller-manager-d6f4b65b8-q9lp8\" (UID: \"a9bb7c91-9123-4dc4-a469-7b7af81d9d4d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q9lp8" Sep 11 14:46:05.575250 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:46:05.575127 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9bb7c91-9123-4dc4-a469-7b7af81d9d4d-cert\") pod \"kserve-controller-manager-d6f4b65b8-q9lp8\" (UID: \"a9bb7c91-9123-4dc4-a469-7b7af81d9d4d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q9lp8" Sep 11 14:46:05.577454 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:46:05.577435 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9bb7c91-9123-4dc4-a469-7b7af81d9d4d-cert\") pod \"kserve-controller-manager-d6f4b65b8-q9lp8\" (UID: \"a9bb7c91-9123-4dc4-a469-7b7af81d9d4d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q9lp8" Sep 11 14:46:05.585492 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:46:05.585472 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l69kd\" (UniqueName: \"kubernetes.io/projected/a9bb7c91-9123-4dc4-a469-7b7af81d9d4d-kube-api-access-l69kd\") pod \"kserve-controller-manager-d6f4b65b8-q9lp8\" (UID: \"a9bb7c91-9123-4dc4-a469-7b7af81d9d4d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q9lp8" Sep 11 14:46:05.673288 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:46:05.673264 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q9lp8" Sep 11 14:46:05.955654 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:46:05.955580 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-q9lp8"] Sep 11 14:46:05.959373 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:46:05.959344 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9bb7c91_9123_4dc4_a469_7b7af81d9d4d.slice/crio-f7f0034e3daa459bd55fd1230c1f7832f43d96b8433571ba36ee48fe5105916d WatchSource:0}: Error finding container f7f0034e3daa459bd55fd1230c1f7832f43d96b8433571ba36ee48fe5105916d: Status 404 returned error can't find the container with id f7f0034e3daa459bd55fd1230c1f7832f43d96b8433571ba36ee48fe5105916d Sep 11 14:46:06.803484 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:46:06.803445 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q9lp8" event={"ID":"a9bb7c91-9123-4dc4-a469-7b7af81d9d4d","Type":"ContainerStarted","Data":"f7f0034e3daa459bd55fd1230c1f7832f43d96b8433571ba36ee48fe5105916d"} Sep 11 14:46:08.814947 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:46:08.814913 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q9lp8" event={"ID":"a9bb7c91-9123-4dc4-a469-7b7af81d9d4d","Type":"ContainerStarted","Data":"4bedfb140e2b12cce72d2ee03312199d1caaff9b98337b8ee3f3a2641058ad65"} Sep 11 14:46:08.815334 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:46:08.815034 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q9lp8" Sep 11 14:46:08.846089 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:46:08.846044 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q9lp8" podStartSLOduration=1.831625405 podStartE2EDuration="3.846027543s" totalImagesPullingTime="2.014402138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:46:08.84342021 +0000 UTC m=+360.087405866" watchObservedRunningTime="2026-09-11 14:46:08.846027543 +0000 UTC m=+360.090013209" Sep 11 14:46:11.780652 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:46:11.780617 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-r846f" Sep 11 14:46:11.782736 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:46:11.782715 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-s7722" Sep 11 14:46:39.824956 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:46:39.824927 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q9lp8" Sep 11 14:46:49.091077 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:46:49.091044 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bc9d7ddcc-zzvvs"] Sep 11 14:47:00.016770 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.016738 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf"] Sep 11 14:47:00.022223 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.022206 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf" Sep 11 14:47:00.025068 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.025043 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 14:47:00.025168 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.025089 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 14:47:00.025819 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.025797 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lvjqc\"" Sep 11 14:47:00.037124 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.037099 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf"] Sep 11 14:47:00.087389 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.087368 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwpxl\" (UniqueName: \"kubernetes.io/projected/7b69b174-a6e2-4333-b773-035bd85a4d0e-kube-api-access-cwpxl\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf\" (UID: \"7b69b174-a6e2-4333-b773-035bd85a4d0e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf" Sep 11 14:47:00.087489 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.087401 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b69b174-a6e2-4333-b773-035bd85a4d0e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf\" (UID: \"7b69b174-a6e2-4333-b773-035bd85a4d0e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf" Sep 11 14:47:00.087489 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.087442 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b69b174-a6e2-4333-b773-035bd85a4d0e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf\" (UID: \"7b69b174-a6e2-4333-b773-035bd85a4d0e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf" Sep 11 14:47:00.120634 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.120611 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj"] Sep 11 14:47:00.124396 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.124381 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj" Sep 11 14:47:00.135866 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.135845 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj"] Sep 11 14:47:00.188041 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.188011 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b69b174-a6e2-4333-b773-035bd85a4d0e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf\" (UID: \"7b69b174-a6e2-4333-b773-035bd85a4d0e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf" Sep 11 14:47:00.188157 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.188049 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/320d8efb-9a42-4fb4-a43a-67766f635701-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj\" (UID: \"320d8efb-9a42-4fb4-a43a-67766f635701\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj" Sep 11 14:47:00.188157 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.188089 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwpxl\" (UniqueName: \"kubernetes.io/projected/7b69b174-a6e2-4333-b773-035bd85a4d0e-kube-api-access-cwpxl\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf\" (UID: \"7b69b174-a6e2-4333-b773-035bd85a4d0e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf" Sep 11 14:47:00.188157 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.188105 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg5r8\" (UniqueName: \"kubernetes.io/projected/320d8efb-9a42-4fb4-a43a-67766f635701-kube-api-access-tg5r8\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj\" (UID: \"320d8efb-9a42-4fb4-a43a-67766f635701\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj" Sep 11 14:47:00.188157 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.188126 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b69b174-a6e2-4333-b773-035bd85a4d0e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf\" (UID: \"7b69b174-a6e2-4333-b773-035bd85a4d0e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf" Sep 11 14:47:00.188157 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.188141 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/320d8efb-9a42-4fb4-a43a-67766f635701-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj\" (UID: \"320d8efb-9a42-4fb4-a43a-67766f635701\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj" Sep 11 14:47:00.188440 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.188378 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b69b174-a6e2-4333-b773-035bd85a4d0e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf\" (UID: \"7b69b174-a6e2-4333-b773-035bd85a4d0e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf" Sep 11 14:47:00.188500 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.188465 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b69b174-a6e2-4333-b773-035bd85a4d0e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf\" (UID: \"7b69b174-a6e2-4333-b773-035bd85a4d0e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf" Sep 11 14:47:00.196521 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.196502 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwpxl\" (UniqueName: \"kubernetes.io/projected/7b69b174-a6e2-4333-b773-035bd85a4d0e-kube-api-access-cwpxl\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf\" (UID: \"7b69b174-a6e2-4333-b773-035bd85a4d0e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf" Sep 11 14:47:00.213588 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.213563 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj"] Sep 11 14:47:00.217426 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.217408 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj" Sep 11 14:47:00.262406 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.262386 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj"] Sep 11 14:47:00.289494 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.289438 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b6d34cb-29be-436c-afb4-03fdf604bd47-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj\" (UID: \"1b6d34cb-29be-436c-afb4-03fdf604bd47\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj" Sep 11 14:47:00.289494 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.289471 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/320d8efb-9a42-4fb4-a43a-67766f635701-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj\" (UID: \"320d8efb-9a42-4fb4-a43a-67766f635701\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj" Sep 11 14:47:00.289661 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.289500 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b6d34cb-29be-436c-afb4-03fdf604bd47-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj\" (UID: \"1b6d34cb-29be-436c-afb4-03fdf604bd47\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj" Sep 11 14:47:00.289661 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.289558 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg5r8\" (UniqueName: \"kubernetes.io/projected/320d8efb-9a42-4fb4-a43a-67766f635701-kube-api-access-tg5r8\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj\" (UID: \"320d8efb-9a42-4fb4-a43a-67766f635701\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj" Sep 11 14:47:00.289661 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.289594 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/320d8efb-9a42-4fb4-a43a-67766f635701-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj\" (UID: \"320d8efb-9a42-4fb4-a43a-67766f635701\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj" Sep 11 14:47:00.289661 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.289623 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6rvh\" (UniqueName: \"kubernetes.io/projected/1b6d34cb-29be-436c-afb4-03fdf604bd47-kube-api-access-r6rvh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj\" (UID: \"1b6d34cb-29be-436c-afb4-03fdf604bd47\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj" Sep 11 14:47:00.289905 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.289886 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/320d8efb-9a42-4fb4-a43a-67766f635701-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj\" (UID: \"320d8efb-9a42-4fb4-a43a-67766f635701\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj" Sep 11 14:47:00.289951 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.289909 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/320d8efb-9a42-4fb4-a43a-67766f635701-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj\" (UID: \"320d8efb-9a42-4fb4-a43a-67766f635701\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj" Sep 11 14:47:00.297974 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.297954 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg5r8\" (UniqueName: \"kubernetes.io/projected/320d8efb-9a42-4fb4-a43a-67766f635701-kube-api-access-tg5r8\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj\" (UID: \"320d8efb-9a42-4fb4-a43a-67766f635701\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj" Sep 11 14:47:00.330446 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.330412 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj"] Sep 11 14:47:00.332819 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.332797 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf" Sep 11 14:47:00.334011 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.333998 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj" Sep 11 14:47:00.343139 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.343118 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj"] Sep 11 14:47:00.390288 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.390253 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vp5b\" (UniqueName: \"kubernetes.io/projected/d5a85de0-7627-4974-ae36-29278c23b056-kube-api-access-8vp5b\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj\" (UID: \"d5a85de0-7627-4974-ae36-29278c23b056\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj" Sep 11 14:47:00.390396 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.390307 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b6d34cb-29be-436c-afb4-03fdf604bd47-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj\" (UID: \"1b6d34cb-29be-436c-afb4-03fdf604bd47\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj" Sep 11 14:47:00.390464 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.390399 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b6d34cb-29be-436c-afb4-03fdf604bd47-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj\" (UID: \"1b6d34cb-29be-436c-afb4-03fdf604bd47\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj" Sep 11 14:47:00.390522 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.390494 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5a85de0-7627-4974-ae36-29278c23b056-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj\" (UID: \"d5a85de0-7627-4974-ae36-29278c23b056\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj" Sep 11 14:47:00.390575 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.390533 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5a85de0-7627-4974-ae36-29278c23b056-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj\" (UID: \"d5a85de0-7627-4974-ae36-29278c23b056\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj" Sep 11 14:47:00.390626 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.390568 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6rvh\" (UniqueName: \"kubernetes.io/projected/1b6d34cb-29be-436c-afb4-03fdf604bd47-kube-api-access-r6rvh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj\" (UID: \"1b6d34cb-29be-436c-afb4-03fdf604bd47\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj" Sep 11 14:47:00.391756 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.391279 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b6d34cb-29be-436c-afb4-03fdf604bd47-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj\" (UID: \"1b6d34cb-29be-436c-afb4-03fdf604bd47\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj" Sep 11 14:47:00.391756 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.391697 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b6d34cb-29be-436c-afb4-03fdf604bd47-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj\" (UID: \"1b6d34cb-29be-436c-afb4-03fdf604bd47\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj" Sep 11 14:47:00.400296 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.400263 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6rvh\" (UniqueName: \"kubernetes.io/projected/1b6d34cb-29be-436c-afb4-03fdf604bd47-kube-api-access-r6rvh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj\" (UID: \"1b6d34cb-29be-436c-afb4-03fdf604bd47\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj" Sep 11 14:47:00.432387 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.432367 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj" Sep 11 14:47:00.491270 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.491239 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vp5b\" (UniqueName: \"kubernetes.io/projected/d5a85de0-7627-4974-ae36-29278c23b056-kube-api-access-8vp5b\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj\" (UID: \"d5a85de0-7627-4974-ae36-29278c23b056\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj" Sep 11 14:47:00.491383 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.491351 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5a85de0-7627-4974-ae36-29278c23b056-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj\" (UID: \"d5a85de0-7627-4974-ae36-29278c23b056\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj" Sep 11 14:47:00.491987 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.491957 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5a85de0-7627-4974-ae36-29278c23b056-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj\" (UID: \"d5a85de0-7627-4974-ae36-29278c23b056\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj" Sep 11 14:47:00.492257 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.492236 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5a85de0-7627-4974-ae36-29278c23b056-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj\" (UID: \"d5a85de0-7627-4974-ae36-29278c23b056\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj" Sep 11 14:47:00.492381 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.492361 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5a85de0-7627-4974-ae36-29278c23b056-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj\" (UID: \"d5a85de0-7627-4974-ae36-29278c23b056\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj" Sep 11 14:47:00.506506 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.506485 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vp5b\" (UniqueName: \"kubernetes.io/projected/d5a85de0-7627-4974-ae36-29278c23b056-kube-api-access-8vp5b\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj\" (UID: \"d5a85de0-7627-4974-ae36-29278c23b056\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj" Sep 11 14:47:00.526297 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.526273 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj" Sep 11 14:47:00.611666 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.611636 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf"] Sep 11 14:47:00.613312 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:47:00.613290 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b69b174_a6e2_4333_b773_035bd85a4d0e.slice/crio-170cb65f9b292ba9a0e1209c22ee62c7eade113a8ed6ca78456b11bed8077138 WatchSource:0}: Error finding container 170cb65f9b292ba9a0e1209c22ee62c7eade113a8ed6ca78456b11bed8077138: Status 404 returned error can't find the container with id 170cb65f9b292ba9a0e1209c22ee62c7eade113a8ed6ca78456b11bed8077138 Sep 11 14:47:00.655891 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.655861 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj" Sep 11 14:47:00.712858 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.712397 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj"] Sep 11 14:47:00.716126 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:47:00.716072 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod320d8efb_9a42_4fb4_a43a_67766f635701.slice/crio-a0151bc57dc9d48368fec89a8ca80c39c9e5abbd803c30a9d3c134f890755b92 WatchSource:0}: Error finding container a0151bc57dc9d48368fec89a8ca80c39c9e5abbd803c30a9d3c134f890755b92: Status 404 returned error can't find the container with id a0151bc57dc9d48368fec89a8ca80c39c9e5abbd803c30a9d3c134f890755b92 Sep 11 14:47:00.809761 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.809737 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj"] Sep 11 14:47:00.811929 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:47:00.811902 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6d34cb_29be_436c_afb4_03fdf604bd47.slice/crio-55abfad227891ccb510a1afa790ffd31d454b4446aaac0d67b363522e0dde22c WatchSource:0}: Error finding container 55abfad227891ccb510a1afa790ffd31d454b4446aaac0d67b363522e0dde22c: Status 404 returned error can't find the container with id 55abfad227891ccb510a1afa790ffd31d454b4446aaac0d67b363522e0dde22c Sep 11 14:47:00.935504 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:00.935473 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj"] Sep 11 14:47:00.938340 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:47:00.938302 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a85de0_7627_4974_ae36_29278c23b056.slice/crio-e05eb5c3c86f7711bcfb22178a2510d353ede1ce25e45d1ca6a41d2527362b84 WatchSource:0}: Error finding container e05eb5c3c86f7711bcfb22178a2510d353ede1ce25e45d1ca6a41d2527362b84: Status 404 returned error can't find the container with id e05eb5c3c86f7711bcfb22178a2510d353ede1ce25e45d1ca6a41d2527362b84 Sep 11 14:47:01.042619 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:01.042593 2583 generic.go:356] "Generic (PLEG): container finished" podID="1b6d34cb-29be-436c-afb4-03fdf604bd47" containerID="767c205f707a52a84f9624d50dc9d77b800846f308f8b723f0cac55ed2655cef" exitCode=0 Sep 11 14:47:01.042964 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:01.042656 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj" event={"ID":"1b6d34cb-29be-436c-afb4-03fdf604bd47","Type":"ContainerDied","Data":"767c205f707a52a84f9624d50dc9d77b800846f308f8b723f0cac55ed2655cef"} Sep 11 14:47:01.042964 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:01.042726 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj" event={"ID":"1b6d34cb-29be-436c-afb4-03fdf604bd47","Type":"ContainerStarted","Data":"55abfad227891ccb510a1afa790ffd31d454b4446aaac0d67b363522e0dde22c"} Sep 11 14:47:01.044001 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:01.043977 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj" event={"ID":"d5a85de0-7627-4974-ae36-29278c23b056","Type":"ContainerStarted","Data":"e05eb5c3c86f7711bcfb22178a2510d353ede1ce25e45d1ca6a41d2527362b84"} Sep 11 14:47:01.045466 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:01.045448 2583 generic.go:356] "Generic (PLEG): container finished" podID="320d8efb-9a42-4fb4-a43a-67766f635701" containerID="2a9707d9eec810f6438b4484e66d426f037c5557ec16e0b92a231f816efa64d5" exitCode=0 Sep 11 14:47:01.045542 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:01.045512 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj" event={"ID":"320d8efb-9a42-4fb4-a43a-67766f635701","Type":"ContainerDied","Data":"2a9707d9eec810f6438b4484e66d426f037c5557ec16e0b92a231f816efa64d5"} Sep 11 14:47:01.045597 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:01.045540 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj" event={"ID":"320d8efb-9a42-4fb4-a43a-67766f635701","Type":"ContainerStarted","Data":"a0151bc57dc9d48368fec89a8ca80c39c9e5abbd803c30a9d3c134f890755b92"} Sep 11 14:47:01.047314 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:01.047293 2583 generic.go:356] "Generic (PLEG): container finished" podID="7b69b174-a6e2-4333-b773-035bd85a4d0e" containerID="d661af63e0ad42fc85390cc1eb81c19853b9e5708079f1c77088ff826c70be47" exitCode=0 Sep 11 14:47:01.047386 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:01.047329 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf" event={"ID":"7b69b174-a6e2-4333-b773-035bd85a4d0e","Type":"ContainerDied","Data":"d661af63e0ad42fc85390cc1eb81c19853b9e5708079f1c77088ff826c70be47"} Sep 11 14:47:01.047386 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:01.047354 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf" event={"ID":"7b69b174-a6e2-4333-b773-035bd85a4d0e","Type":"ContainerStarted","Data":"170cb65f9b292ba9a0e1209c22ee62c7eade113a8ed6ca78456b11bed8077138"} Sep 11 14:47:02.054144 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:02.054116 2583 generic.go:356] "Generic (PLEG): container finished" podID="1b6d34cb-29be-436c-afb4-03fdf604bd47" containerID="73a3758a1b557b808257f9060032de20ddd03b3c9b46938854313f2a1b07860a" exitCode=0 Sep 11 14:47:02.054500 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:02.054206 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj" event={"ID":"1b6d34cb-29be-436c-afb4-03fdf604bd47","Type":"ContainerDied","Data":"73a3758a1b557b808257f9060032de20ddd03b3c9b46938854313f2a1b07860a"} Sep 11 14:47:02.055766 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:02.055746 2583 generic.go:356] "Generic (PLEG): container finished" podID="d5a85de0-7627-4974-ae36-29278c23b056" containerID="29bb39b7556961ee8813eb90031e06c81f49a7d670e15f64360b0c211309ae2e" exitCode=0 Sep 11 14:47:02.055865 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:02.055817 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj" event={"ID":"d5a85de0-7627-4974-ae36-29278c23b056","Type":"ContainerDied","Data":"29bb39b7556961ee8813eb90031e06c81f49a7d670e15f64360b0c211309ae2e"} Sep 11 14:47:03.063205 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:03.063178 2583 generic.go:356] "Generic (PLEG): container finished" podID="1b6d34cb-29be-436c-afb4-03fdf604bd47" containerID="8ba4c4cfc93fa298c804461afb60c1c2521c84a5b30582d6fd5bce453e7888b9" exitCode=0 Sep 11 14:47:03.063537 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:03.063255 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj" event={"ID":"1b6d34cb-29be-436c-afb4-03fdf604bd47","Type":"ContainerDied","Data":"8ba4c4cfc93fa298c804461afb60c1c2521c84a5b30582d6fd5bce453e7888b9"} Sep 11 14:47:03.064947 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:03.064928 2583 generic.go:356] "Generic (PLEG): container finished" podID="320d8efb-9a42-4fb4-a43a-67766f635701" containerID="3ebca39bdd697c19180efb56a5294176103ecbc88cde444c2c455d26f0e51e67" exitCode=0 Sep 11 14:47:03.065020 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:03.064998 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj" event={"ID":"320d8efb-9a42-4fb4-a43a-67766f635701","Type":"ContainerDied","Data":"3ebca39bdd697c19180efb56a5294176103ecbc88cde444c2c455d26f0e51e67"} Sep 11 14:47:03.066746 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:03.066729 2583 generic.go:356] "Generic (PLEG): container finished" podID="7b69b174-a6e2-4333-b773-035bd85a4d0e" containerID="37a1858f4227da4410277a2e59bb52eb48497c019833f51d01cd4f2647d8b12c" exitCode=0 Sep 11 14:47:03.066807 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:03.066757 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf" event={"ID":"7b69b174-a6e2-4333-b773-035bd85a4d0e","Type":"ContainerDied","Data":"37a1858f4227da4410277a2e59bb52eb48497c019833f51d01cd4f2647d8b12c"} Sep 11 14:47:04.073753 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:04.073697 2583 generic.go:356] "Generic (PLEG): container finished" podID="7b69b174-a6e2-4333-b773-035bd85a4d0e" containerID="4d50d31ccc9b62b2c9d88651817f5ad4bd87dd49df1336bc27e221ba0adcb4a8" exitCode=0 Sep 11 14:47:04.074176 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:04.073802 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf" event={"ID":"7b69b174-a6e2-4333-b773-035bd85a4d0e","Type":"ContainerDied","Data":"4d50d31ccc9b62b2c9d88651817f5ad4bd87dd49df1336bc27e221ba0adcb4a8"} Sep 11 14:47:04.075778 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:04.075743 2583 generic.go:356] "Generic (PLEG): container finished" podID="d5a85de0-7627-4974-ae36-29278c23b056" containerID="3ddcaaa860975e30d08516deaa84ab4aea6b12627eedbaa734ec6b9cf82357b7" exitCode=0 Sep 11 14:47:04.075922 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:04.075791 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj" event={"ID":"d5a85de0-7627-4974-ae36-29278c23b056","Type":"ContainerDied","Data":"3ddcaaa860975e30d08516deaa84ab4aea6b12627eedbaa734ec6b9cf82357b7"} Sep 11 14:47:04.078233 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:04.078208 2583 generic.go:356] "Generic (PLEG): container finished" podID="320d8efb-9a42-4fb4-a43a-67766f635701" containerID="aa34e18eb8f9df6008f53de11020e49f93cb3f2b5abc0e75dcbad016772aca5d" exitCode=0 Sep 11 14:47:04.078338 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:04.078298 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj" event={"ID":"320d8efb-9a42-4fb4-a43a-67766f635701","Type":"ContainerDied","Data":"aa34e18eb8f9df6008f53de11020e49f93cb3f2b5abc0e75dcbad016772aca5d"} Sep 11 14:47:04.206126 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:04.206099 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj" Sep 11 14:47:04.221014 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:04.220994 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b6d34cb-29be-436c-afb4-03fdf604bd47-util\") pod \"1b6d34cb-29be-436c-afb4-03fdf604bd47\" (UID: \"1b6d34cb-29be-436c-afb4-03fdf604bd47\") " Sep 11 14:47:04.221112 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:04.221045 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b6d34cb-29be-436c-afb4-03fdf604bd47-bundle\") pod \"1b6d34cb-29be-436c-afb4-03fdf604bd47\" (UID: \"1b6d34cb-29be-436c-afb4-03fdf604bd47\") " Sep 11 14:47:04.221112 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:04.221103 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6rvh\" (UniqueName: \"kubernetes.io/projected/1b6d34cb-29be-436c-afb4-03fdf604bd47-kube-api-access-r6rvh\") pod \"1b6d34cb-29be-436c-afb4-03fdf604bd47\" (UID: \"1b6d34cb-29be-436c-afb4-03fdf604bd47\") " Sep 11 14:47:04.221561 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:04.221529 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6d34cb-29be-436c-afb4-03fdf604bd47-bundle" (OuterVolumeSpecName: "bundle") pod "1b6d34cb-29be-436c-afb4-03fdf604bd47" (UID: "1b6d34cb-29be-436c-afb4-03fdf604bd47"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:47:04.223285 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:04.223261 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b6d34cb-29be-436c-afb4-03fdf604bd47-kube-api-access-r6rvh" (OuterVolumeSpecName: "kube-api-access-r6rvh") pod "1b6d34cb-29be-436c-afb4-03fdf604bd47" (UID: "1b6d34cb-29be-436c-afb4-03fdf604bd47"). InnerVolumeSpecName "kube-api-access-r6rvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:47:04.226356 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:04.226335 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6d34cb-29be-436c-afb4-03fdf604bd47-util" (OuterVolumeSpecName: "util") pod "1b6d34cb-29be-436c-afb4-03fdf604bd47" (UID: "1b6d34cb-29be-436c-afb4-03fdf604bd47"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:47:04.321840 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:04.321814 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b6d34cb-29be-436c-afb4-03fdf604bd47-util\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:47:04.321840 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:04.321836 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b6d34cb-29be-436c-afb4-03fdf604bd47-bundle\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:47:04.321956 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:04.321844 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6rvh\" (UniqueName: \"kubernetes.io/projected/1b6d34cb-29be-436c-afb4-03fdf604bd47-kube-api-access-r6rvh\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:47:05.085752 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:05.085724 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj" Sep 11 14:47:05.086180 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:05.085727 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fmndj" event={"ID":"1b6d34cb-29be-436c-afb4-03fdf604bd47","Type":"ContainerDied","Data":"55abfad227891ccb510a1afa790ffd31d454b4446aaac0d67b363522e0dde22c"} Sep 11 14:47:05.086180 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:05.085841 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55abfad227891ccb510a1afa790ffd31d454b4446aaac0d67b363522e0dde22c" Sep 11 14:47:05.087985 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:05.087962 2583 generic.go:356] "Generic (PLEG): container finished" podID="d5a85de0-7627-4974-ae36-29278c23b056" containerID="72b21cf623ef0fd39c38682178df3ceac435dc9062486138791aced4c65cd47f" exitCode=0 Sep 11 14:47:05.088119 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:05.088044 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj" event={"ID":"d5a85de0-7627-4974-ae36-29278c23b056","Type":"ContainerDied","Data":"72b21cf623ef0fd39c38682178df3ceac435dc9062486138791aced4c65cd47f"} Sep 11 14:47:05.215432 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:05.215406 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf" Sep 11 14:47:05.248905 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:05.248878 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj" Sep 11 14:47:05.330613 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:05.330582 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b69b174-a6e2-4333-b773-035bd85a4d0e-util\") pod \"7b69b174-a6e2-4333-b773-035bd85a4d0e\" (UID: \"7b69b174-a6e2-4333-b773-035bd85a4d0e\") " Sep 11 14:47:05.330788 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:05.330648 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tg5r8\" (UniqueName: \"kubernetes.io/projected/320d8efb-9a42-4fb4-a43a-67766f635701-kube-api-access-tg5r8\") pod \"320d8efb-9a42-4fb4-a43a-67766f635701\" (UID: \"320d8efb-9a42-4fb4-a43a-67766f635701\") " Sep 11 14:47:05.330788 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:05.330718 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/320d8efb-9a42-4fb4-a43a-67766f635701-bundle\") pod \"320d8efb-9a42-4fb4-a43a-67766f635701\" (UID: \"320d8efb-9a42-4fb4-a43a-67766f635701\") " Sep 11 14:47:05.330788 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:05.330737 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/320d8efb-9a42-4fb4-a43a-67766f635701-util\") pod \"320d8efb-9a42-4fb4-a43a-67766f635701\" (UID: \"320d8efb-9a42-4fb4-a43a-67766f635701\") " Sep 11 14:47:05.330951 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:05.330798 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwpxl\" (UniqueName: \"kubernetes.io/projected/7b69b174-a6e2-4333-b773-035bd85a4d0e-kube-api-access-cwpxl\") pod \"7b69b174-a6e2-4333-b773-035bd85a4d0e\" (UID: \"7b69b174-a6e2-4333-b773-035bd85a4d0e\") " Sep 11 14:47:05.330951 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:05.330831 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b69b174-a6e2-4333-b773-035bd85a4d0e-bundle\") pod \"7b69b174-a6e2-4333-b773-035bd85a4d0e\" (UID: \"7b69b174-a6e2-4333-b773-035bd85a4d0e\") " Sep 11 14:47:05.331581 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:05.331544 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320d8efb-9a42-4fb4-a43a-67766f635701-bundle" (OuterVolumeSpecName: "bundle") pod "320d8efb-9a42-4fb4-a43a-67766f635701" (UID: "320d8efb-9a42-4fb4-a43a-67766f635701"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:47:05.331947 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:05.331921 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b69b174-a6e2-4333-b773-035bd85a4d0e-bundle" (OuterVolumeSpecName: "bundle") pod "7b69b174-a6e2-4333-b773-035bd85a4d0e" (UID: "7b69b174-a6e2-4333-b773-035bd85a4d0e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:47:05.332918 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:05.332886 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320d8efb-9a42-4fb4-a43a-67766f635701-kube-api-access-tg5r8" (OuterVolumeSpecName: "kube-api-access-tg5r8") pod "320d8efb-9a42-4fb4-a43a-67766f635701" (UID: "320d8efb-9a42-4fb4-a43a-67766f635701"). InnerVolumeSpecName "kube-api-access-tg5r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:47:05.333174 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:05.333156 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b69b174-a6e2-4333-b773-035bd85a4d0e-kube-api-access-cwpxl" (OuterVolumeSpecName: "kube-api-access-cwpxl") pod "7b69b174-a6e2-4333-b773-035bd85a4d0e" (UID: "7b69b174-a6e2-4333-b773-035bd85a4d0e"). InnerVolumeSpecName "kube-api-access-cwpxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:47:05.337205 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:05.337137 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b69b174-a6e2-4333-b773-035bd85a4d0e-util" (OuterVolumeSpecName: "util") pod "7b69b174-a6e2-4333-b773-035bd85a4d0e" (UID: "7b69b174-a6e2-4333-b773-035bd85a4d0e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:47:05.337205 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:05.337189 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320d8efb-9a42-4fb4-a43a-67766f635701-util" (OuterVolumeSpecName: "util") pod "320d8efb-9a42-4fb4-a43a-67766f635701" (UID: "320d8efb-9a42-4fb4-a43a-67766f635701"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:47:05.431799 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:05.431775 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwpxl\" (UniqueName: \"kubernetes.io/projected/7b69b174-a6e2-4333-b773-035bd85a4d0e-kube-api-access-cwpxl\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:47:05.431799 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:05.431797 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b69b174-a6e2-4333-b773-035bd85a4d0e-bundle\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:47:05.431909 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:05.431807 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b69b174-a6e2-4333-b773-035bd85a4d0e-util\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:47:05.431909 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:05.431815 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg5r8\" (UniqueName: \"kubernetes.io/projected/320d8efb-9a42-4fb4-a43a-67766f635701-kube-api-access-tg5r8\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:47:05.431909 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:05.431824 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/320d8efb-9a42-4fb4-a43a-67766f635701-bundle\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:47:05.431909 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:05.431831 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/320d8efb-9a42-4fb4-a43a-67766f635701-util\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:47:06.094934 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:06.094903 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj" Sep 11 14:47:06.095383 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:06.094900 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apsljj" event={"ID":"320d8efb-9a42-4fb4-a43a-67766f635701","Type":"ContainerDied","Data":"a0151bc57dc9d48368fec89a8ca80c39c9e5abbd803c30a9d3c134f890755b92"} Sep 11 14:47:06.095383 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:06.095055 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0151bc57dc9d48368fec89a8ca80c39c9e5abbd803c30a9d3c134f890755b92" Sep 11 14:47:06.096858 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:06.096826 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf" event={"ID":"7b69b174-a6e2-4333-b773-035bd85a4d0e","Type":"ContainerDied","Data":"170cb65f9b292ba9a0e1209c22ee62c7eade113a8ed6ca78456b11bed8077138"} Sep 11 14:47:06.096964 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:06.096874 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wlrqf" Sep 11 14:47:06.096964 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:06.096877 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="170cb65f9b292ba9a0e1209c22ee62c7eade113a8ed6ca78456b11bed8077138" Sep 11 14:47:06.217805 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:06.217785 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj" Sep 11 14:47:06.238263 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:06.238220 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5a85de0-7627-4974-ae36-29278c23b056-util\") pod \"d5a85de0-7627-4974-ae36-29278c23b056\" (UID: \"d5a85de0-7627-4974-ae36-29278c23b056\") " Sep 11 14:47:06.238532 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:06.238512 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vp5b\" (UniqueName: \"kubernetes.io/projected/d5a85de0-7627-4974-ae36-29278c23b056-kube-api-access-8vp5b\") pod \"d5a85de0-7627-4974-ae36-29278c23b056\" (UID: \"d5a85de0-7627-4974-ae36-29278c23b056\") " Sep 11 14:47:06.238665 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:06.238652 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5a85de0-7627-4974-ae36-29278c23b056-bundle\") pod \"d5a85de0-7627-4974-ae36-29278c23b056\" (UID: \"d5a85de0-7627-4974-ae36-29278c23b056\") " Sep 11 14:47:06.239446 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:06.239415 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a85de0-7627-4974-ae36-29278c23b056-bundle" (OuterVolumeSpecName: "bundle") pod "d5a85de0-7627-4974-ae36-29278c23b056" (UID: "d5a85de0-7627-4974-ae36-29278c23b056"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:47:06.241170 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:06.241149 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a85de0-7627-4974-ae36-29278c23b056-kube-api-access-8vp5b" (OuterVolumeSpecName: "kube-api-access-8vp5b") pod "d5a85de0-7627-4974-ae36-29278c23b056" (UID: "d5a85de0-7627-4974-ae36-29278c23b056"). InnerVolumeSpecName "kube-api-access-8vp5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:47:06.298425 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:06.298397 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a85de0-7627-4974-ae36-29278c23b056-util" (OuterVolumeSpecName: "util") pod "d5a85de0-7627-4974-ae36-29278c23b056" (UID: "d5a85de0-7627-4974-ae36-29278c23b056"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:47:06.339735 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:06.339692 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5a85de0-7627-4974-ae36-29278c23b056-util\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:47:06.339735 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:06.339735 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vp5b\" (UniqueName: \"kubernetes.io/projected/d5a85de0-7627-4974-ae36-29278c23b056-kube-api-access-8vp5b\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:47:06.339853 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:06.339746 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5a85de0-7627-4974-ae36-29278c23b056-bundle\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:47:07.103078 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:07.103053 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj" Sep 11 14:47:07.103422 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:07.103053 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm5hhj" event={"ID":"d5a85de0-7627-4974-ae36-29278c23b056","Type":"ContainerDied","Data":"e05eb5c3c86f7711bcfb22178a2510d353ede1ce25e45d1ca6a41d2527362b84"} Sep 11 14:47:07.103422 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:07.103171 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e05eb5c3c86f7711bcfb22178a2510d353ede1ce25e45d1ca6a41d2527362b84" Sep 11 14:47:13.896617 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.896581 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pxbzq"] Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.896874 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b69b174-a6e2-4333-b773-035bd85a4d0e" containerName="pull" Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.896885 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b69b174-a6e2-4333-b773-035bd85a4d0e" containerName="pull" Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.896893 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b6d34cb-29be-436c-afb4-03fdf604bd47" containerName="pull" Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.896898 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b6d34cb-29be-436c-afb4-03fdf604bd47" containerName="pull" Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.896906 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="320d8efb-9a42-4fb4-a43a-67766f635701" containerName="util" Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.896911 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="320d8efb-9a42-4fb4-a43a-67766f635701" containerName="util" Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.896920 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="320d8efb-9a42-4fb4-a43a-67766f635701" containerName="extract" Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.896925 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="320d8efb-9a42-4fb4-a43a-67766f635701" containerName="extract" Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.896931 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b6d34cb-29be-436c-afb4-03fdf604bd47" containerName="util" Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.896936 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b6d34cb-29be-436c-afb4-03fdf604bd47" containerName="util" Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.896944 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b69b174-a6e2-4333-b773-035bd85a4d0e" containerName="util" Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.896948 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b69b174-a6e2-4333-b773-035bd85a4d0e" containerName="util" Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.896958 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b69b174-a6e2-4333-b773-035bd85a4d0e" containerName="extract" Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.896963 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b69b174-a6e2-4333-b773-035bd85a4d0e" containerName="extract" Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.896970 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5a85de0-7627-4974-ae36-29278c23b056" containerName="util" Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.896975 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5a85de0-7627-4974-ae36-29278c23b056" containerName="util" Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.896980 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b6d34cb-29be-436c-afb4-03fdf604bd47" containerName="extract" Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.896984 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b6d34cb-29be-436c-afb4-03fdf604bd47" containerName="extract" Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.896991 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5a85de0-7627-4974-ae36-29278c23b056" containerName="pull" Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.896996 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5a85de0-7627-4974-ae36-29278c23b056" containerName="pull" Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.897002 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="320d8efb-9a42-4fb4-a43a-67766f635701" containerName="pull" Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.897007 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="320d8efb-9a42-4fb4-a43a-67766f635701" containerName="pull" Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.897012 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5a85de0-7627-4974-ae36-29278c23b056" containerName="extract" Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.897017 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5a85de0-7627-4974-ae36-29278c23b056" containerName="extract" Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.897062 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5a85de0-7627-4974-ae36-29278c23b056" containerName="extract" Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.897069 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b6d34cb-29be-436c-afb4-03fdf604bd47" containerName="extract" Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.897076 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="320d8efb-9a42-4fb4-a43a-67766f635701" containerName="extract" Sep 11 14:47:13.897075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.897084 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b69b174-a6e2-4333-b773-035bd85a4d0e" containerName="extract" Sep 11 14:47:13.900315 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.900297 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pxbzq" Sep 11 14:47:13.902812 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.902786 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 11 14:47:13.902937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.902848 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 11 14:47:13.902937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.902859 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 11 14:47:13.902937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.902881 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-mx7lx\"" Sep 11 14:47:13.918229 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:13.918206 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pxbzq"] Sep 11 14:47:14.003140 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.003117 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-566x4\" (UniqueName: \"kubernetes.io/projected/2264c0d5-92e6-432f-8e68-26de9c9c3b5a-kube-api-access-566x4\") pod \"dns-operator-controller-manager-7b84cd7c9c-pxbzq\" (UID: \"2264c0d5-92e6-432f-8e68-26de9c9c3b5a\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pxbzq" Sep 11 14:47:14.104367 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.104340 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-566x4\" (UniqueName: \"kubernetes.io/projected/2264c0d5-92e6-432f-8e68-26de9c9c3b5a-kube-api-access-566x4\") pod \"dns-operator-controller-manager-7b84cd7c9c-pxbzq\" (UID: \"2264c0d5-92e6-432f-8e68-26de9c9c3b5a\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pxbzq" Sep 11 14:47:14.114757 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.114692 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-bc9d7ddcc-zzvvs" podUID="8343026f-e77a-4c2e-b224-99e9cefd5ffa" containerName="console" containerID="cri-o://4d887057c99e6fc189f757349396c5f2fe35b08dc52e196732bb266f4f32731e" gracePeriod=15 Sep 11 14:47:14.118768 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.118744 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-566x4\" (UniqueName: \"kubernetes.io/projected/2264c0d5-92e6-432f-8e68-26de9c9c3b5a-kube-api-access-566x4\") pod \"dns-operator-controller-manager-7b84cd7c9c-pxbzq\" (UID: \"2264c0d5-92e6-432f-8e68-26de9c9c3b5a\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pxbzq" Sep 11 14:47:14.209954 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.209899 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pxbzq" Sep 11 14:47:14.377796 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.377771 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:47:14.502954 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.502931 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pxbzq"] Sep 11 14:47:14.505494 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:47:14.505464 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2264c0d5_92e6_432f_8e68_26de9c9c3b5a.slice/crio-8a3624cd0702bcd8202634407ffc6c104b7158034fdc53fab1666448eb43690d WatchSource:0}: Error finding container 8a3624cd0702bcd8202634407ffc6c104b7158034fdc53fab1666448eb43690d: Status 404 returned error can't find the container with id 8a3624cd0702bcd8202634407ffc6c104b7158034fdc53fab1666448eb43690d Sep 11 14:47:14.506516 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.506497 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8343026f-e77a-4c2e-b224-99e9cefd5ffa-trusted-ca-bundle\") pod \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " Sep 11 14:47:14.506623 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.506543 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8343026f-e77a-4c2e-b224-99e9cefd5ffa-console-serving-cert\") pod \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " Sep 11 14:47:14.506623 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.506575 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8343026f-e77a-4c2e-b224-99e9cefd5ffa-service-ca\") pod \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " Sep 11 14:47:14.506623 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.506606 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mmlz\" (UniqueName: \"kubernetes.io/projected/8343026f-e77a-4c2e-b224-99e9cefd5ffa-kube-api-access-7mmlz\") pod \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " Sep 11 14:47:14.506800 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.506746 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8343026f-e77a-4c2e-b224-99e9cefd5ffa-console-config\") pod \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " Sep 11 14:47:14.506859 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.506826 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8343026f-e77a-4c2e-b224-99e9cefd5ffa-console-oauth-config\") pod \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " Sep 11 14:47:14.506911 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.506859 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8343026f-e77a-4c2e-b224-99e9cefd5ffa-oauth-serving-cert\") pod \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\" (UID: \"8343026f-e77a-4c2e-b224-99e9cefd5ffa\") " Sep 11 14:47:14.507218 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.507170 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8343026f-e77a-4c2e-b224-99e9cefd5ffa-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8343026f-e77a-4c2e-b224-99e9cefd5ffa" (UID: "8343026f-e77a-4c2e-b224-99e9cefd5ffa"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:47:14.507321 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.507262 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8343026f-e77a-4c2e-b224-99e9cefd5ffa-console-config" (OuterVolumeSpecName: "console-config") pod "8343026f-e77a-4c2e-b224-99e9cefd5ffa" (UID: "8343026f-e77a-4c2e-b224-99e9cefd5ffa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:47:14.507321 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.507285 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8343026f-e77a-4c2e-b224-99e9cefd5ffa-service-ca" (OuterVolumeSpecName: "service-ca") pod "8343026f-e77a-4c2e-b224-99e9cefd5ffa" (UID: "8343026f-e77a-4c2e-b224-99e9cefd5ffa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:47:14.507434 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.507331 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8343026f-e77a-4c2e-b224-99e9cefd5ffa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8343026f-e77a-4c2e-b224-99e9cefd5ffa" (UID: "8343026f-e77a-4c2e-b224-99e9cefd5ffa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:47:14.508932 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.508907 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8343026f-e77a-4c2e-b224-99e9cefd5ffa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8343026f-e77a-4c2e-b224-99e9cefd5ffa" (UID: "8343026f-e77a-4c2e-b224-99e9cefd5ffa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:47:14.509052 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.509035 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8343026f-e77a-4c2e-b224-99e9cefd5ffa-kube-api-access-7mmlz" (OuterVolumeSpecName: "kube-api-access-7mmlz") pod "8343026f-e77a-4c2e-b224-99e9cefd5ffa" (UID: "8343026f-e77a-4c2e-b224-99e9cefd5ffa"). InnerVolumeSpecName "kube-api-access-7mmlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:47:14.509052 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.509060 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8343026f-e77a-4c2e-b224-99e9cefd5ffa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8343026f-e77a-4c2e-b224-99e9cefd5ffa" (UID: "8343026f-e77a-4c2e-b224-99e9cefd5ffa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:47:14.607741 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.607691 2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8343026f-e77a-4c2e-b224-99e9cefd5ffa-console-config\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:47:14.607741 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.607741 2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8343026f-e77a-4c2e-b224-99e9cefd5ffa-console-oauth-config\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:47:14.607918 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.607756 2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8343026f-e77a-4c2e-b224-99e9cefd5ffa-oauth-serving-cert\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:47:14.607918 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.607768 2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8343026f-e77a-4c2e-b224-99e9cefd5ffa-trusted-ca-bundle\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:47:14.607918 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.607779 2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8343026f-e77a-4c2e-b224-99e9cefd5ffa-console-serving-cert\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:47:14.607918 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.607790 2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8343026f-e77a-4c2e-b224-99e9cefd5ffa-service-ca\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:47:14.607918 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:14.607801 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mmlz\" (UniqueName: \"kubernetes.io/projected/8343026f-e77a-4c2e-b224-99e9cefd5ffa-kube-api-access-7mmlz\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:47:15.149288 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:15.149254 2583 generic.go:356] "Generic (PLEG): container finished" podID="8343026f-e77a-4c2e-b224-99e9cefd5ffa" containerID="4d887057c99e6fc189f757349396c5f2fe35b08dc52e196732bb266f4f32731e" exitCode=0 Sep 11 14:47:15.149668 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:15.149302 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bc9d7ddcc-zzvvs" Sep 11 14:47:15.149668 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:15.149346 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bc9d7ddcc-zzvvs" event={"ID":"8343026f-e77a-4c2e-b224-99e9cefd5ffa","Type":"ContainerDied","Data":"4d887057c99e6fc189f757349396c5f2fe35b08dc52e196732bb266f4f32731e"} Sep 11 14:47:15.149668 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:15.149392 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bc9d7ddcc-zzvvs" event={"ID":"8343026f-e77a-4c2e-b224-99e9cefd5ffa","Type":"ContainerDied","Data":"42bb6fe5a0950f4e02aa9355445f15b97f04fef3f37de1fd234a6fec4332fa59"} Sep 11 14:47:15.149668 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:15.149411 2583 scope.go:117] "RemoveContainer" containerID="4d887057c99e6fc189f757349396c5f2fe35b08dc52e196732bb266f4f32731e" Sep 11 14:47:15.150922 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:15.150901 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pxbzq" event={"ID":"2264c0d5-92e6-432f-8e68-26de9c9c3b5a","Type":"ContainerStarted","Data":"8a3624cd0702bcd8202634407ffc6c104b7158034fdc53fab1666448eb43690d"} Sep 11 14:47:15.157881 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:15.157866 2583 scope.go:117] "RemoveContainer" containerID="4d887057c99e6fc189f757349396c5f2fe35b08dc52e196732bb266f4f32731e" Sep 11 14:47:15.158122 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:47:15.158106 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d887057c99e6fc189f757349396c5f2fe35b08dc52e196732bb266f4f32731e\": container with ID starting with 4d887057c99e6fc189f757349396c5f2fe35b08dc52e196732bb266f4f32731e not found: ID does not exist" containerID="4d887057c99e6fc189f757349396c5f2fe35b08dc52e196732bb266f4f32731e" Sep 11 14:47:15.158170 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:15.158129 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d887057c99e6fc189f757349396c5f2fe35b08dc52e196732bb266f4f32731e"} err="failed to get container status \"4d887057c99e6fc189f757349396c5f2fe35b08dc52e196732bb266f4f32731e\": rpc error: code = NotFound desc = could not find container \"4d887057c99e6fc189f757349396c5f2fe35b08dc52e196732bb266f4f32731e\": container with ID starting with 4d887057c99e6fc189f757349396c5f2fe35b08dc52e196732bb266f4f32731e not found: ID does not exist" Sep 11 14:47:15.176813 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:15.176789 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bc9d7ddcc-zzvvs"] Sep 11 14:47:15.189791 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:15.189771 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bc9d7ddcc-zzvvs"] Sep 11 14:47:15.332320 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:15.332291 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8343026f-e77a-4c2e-b224-99e9cefd5ffa" path="/var/lib/kubelet/pods/8343026f-e77a-4c2e-b224-99e9cefd5ffa/volumes" Sep 11 14:47:17.166272 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:17.166246 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pxbzq" event={"ID":"2264c0d5-92e6-432f-8e68-26de9c9c3b5a","Type":"ContainerStarted","Data":"dd7253c3624d769c7a74f3a912aff20dd56277b8f759b6951aac17bcb38f9975"} Sep 11 14:47:17.166540 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:17.166301 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pxbzq" Sep 11 14:47:17.195655 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:17.195613 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pxbzq" podStartSLOduration=1.6042409119999999 podStartE2EDuration="4.195597058s" totalImagesPullingTime="2.591356146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:47:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:47:17.193357291 +0000 UTC m=+428.437342948" watchObservedRunningTime="2026-09-11 14:47:17.195597058 +0000 UTC m=+428.439582715" Sep 11 14:47:23.095161 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:23.095118 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rnh64"] Sep 11 14:47:23.095534 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:23.095434 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8343026f-e77a-4c2e-b224-99e9cefd5ffa" containerName="console" Sep 11 14:47:23.095534 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:23.095445 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8343026f-e77a-4c2e-b224-99e9cefd5ffa" containerName="console" Sep 11 14:47:23.095534 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:23.095508 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8343026f-e77a-4c2e-b224-99e9cefd5ffa" containerName="console" Sep 11 14:47:23.098484 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:23.098451 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rnh64" Sep 11 14:47:23.101736 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:23.101692 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-xlpzr\"" Sep 11 14:47:23.115405 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:23.115375 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rnh64"] Sep 11 14:47:23.176265 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:23.176231 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpjv6\" (UniqueName: \"kubernetes.io/projected/b97b33a5-27aa-4b2d-92e6-9878b4b6915a-kube-api-access-cpjv6\") pod \"limitador-operator-controller-manager-74495c69f8-rnh64\" (UID: \"b97b33a5-27aa-4b2d-92e6-9878b4b6915a\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rnh64" Sep 11 14:47:23.276741 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:23.276688 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpjv6\" (UniqueName: \"kubernetes.io/projected/b97b33a5-27aa-4b2d-92e6-9878b4b6915a-kube-api-access-cpjv6\") pod \"limitador-operator-controller-manager-74495c69f8-rnh64\" (UID: \"b97b33a5-27aa-4b2d-92e6-9878b4b6915a\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rnh64" Sep 11 14:47:23.295210 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:23.295182 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpjv6\" (UniqueName: \"kubernetes.io/projected/b97b33a5-27aa-4b2d-92e6-9878b4b6915a-kube-api-access-cpjv6\") pod \"limitador-operator-controller-manager-74495c69f8-rnh64\" (UID: \"b97b33a5-27aa-4b2d-92e6-9878b4b6915a\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rnh64" Sep 11 14:47:23.409000 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:23.408973 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rnh64" Sep 11 14:47:23.682145 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:23.682121 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rnh64"] Sep 11 14:47:23.684117 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:47:23.684092 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97b33a5_27aa_4b2d_92e6_9878b4b6915a.slice/crio-118610afed13673eff2caf85f0c603242fcd618ad9022e07f11fbe1e7fdadbc7 WatchSource:0}: Error finding container 118610afed13673eff2caf85f0c603242fcd618ad9022e07f11fbe1e7fdadbc7: Status 404 returned error can't find the container with id 118610afed13673eff2caf85f0c603242fcd618ad9022e07f11fbe1e7fdadbc7 Sep 11 14:47:24.198874 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:24.198840 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rnh64" event={"ID":"b97b33a5-27aa-4b2d-92e6-9878b4b6915a","Type":"ContainerStarted","Data":"118610afed13673eff2caf85f0c603242fcd618ad9022e07f11fbe1e7fdadbc7"} Sep 11 14:47:25.206633 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:25.206601 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rnh64" event={"ID":"b97b33a5-27aa-4b2d-92e6-9878b4b6915a","Type":"ContainerStarted","Data":"62344cf9a0565cb2f0771c63930f5291a09a206e2dd99e8f2346b96ec164fab0"} Sep 11 14:47:25.207096 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:25.206732 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rnh64" Sep 11 14:47:25.232243 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:25.232172 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rnh64" podStartSLOduration=0.819011304 podStartE2EDuration="2.232153771s" totalImagesPullingTime="1.413142467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:47:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:47:25.230764137 +0000 UTC m=+436.474749814" watchObservedRunningTime="2026-09-11 14:47:25.232153771 +0000 UTC m=+436.476139430" Sep 11 14:47:27.191351 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:27.191318 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-kl8dk"] Sep 11 14:47:27.194497 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:27.194474 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kl8dk" Sep 11 14:47:27.197471 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:27.197446 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6xbxv\"" Sep 11 14:47:27.197579 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:27.197446 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 11 14:47:27.197579 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:27.197452 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 11 14:47:27.208977 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:27.208949 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-kl8dk"] Sep 11 14:47:27.309418 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:27.309391 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp8hg\" (UniqueName: \"kubernetes.io/projected/70c7f356-3f7a-4b55-b032-4165e1864109-kube-api-access-xp8hg\") pod \"kuadrant-console-plugin-fbdd4f445-kl8dk\" (UID: \"70c7f356-3f7a-4b55-b032-4165e1864109\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kl8dk" Sep 11 14:47:27.309568 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:27.309436 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/70c7f356-3f7a-4b55-b032-4165e1864109-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-kl8dk\" (UID: \"70c7f356-3f7a-4b55-b032-4165e1864109\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kl8dk" Sep 11 14:47:27.309568 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:27.309493 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/70c7f356-3f7a-4b55-b032-4165e1864109-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-kl8dk\" (UID: \"70c7f356-3f7a-4b55-b032-4165e1864109\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kl8dk" Sep 11 14:47:27.410325 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:27.410298 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/70c7f356-3f7a-4b55-b032-4165e1864109-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-kl8dk\" (UID: \"70c7f356-3f7a-4b55-b032-4165e1864109\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kl8dk" Sep 11 14:47:27.410468 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:27.410329 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/70c7f356-3f7a-4b55-b032-4165e1864109-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-kl8dk\" (UID: \"70c7f356-3f7a-4b55-b032-4165e1864109\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kl8dk" Sep 11 14:47:27.410468 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:27.410372 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp8hg\" (UniqueName: \"kubernetes.io/projected/70c7f356-3f7a-4b55-b032-4165e1864109-kube-api-access-xp8hg\") pod \"kuadrant-console-plugin-fbdd4f445-kl8dk\" (UID: \"70c7f356-3f7a-4b55-b032-4165e1864109\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kl8dk" Sep 11 14:47:27.410923 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:27.410901 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/70c7f356-3f7a-4b55-b032-4165e1864109-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-kl8dk\" (UID: \"70c7f356-3f7a-4b55-b032-4165e1864109\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kl8dk" Sep 11 14:47:27.412674 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:27.412652 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/70c7f356-3f7a-4b55-b032-4165e1864109-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-kl8dk\" (UID: \"70c7f356-3f7a-4b55-b032-4165e1864109\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kl8dk" Sep 11 14:47:27.418889 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:27.418859 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp8hg\" (UniqueName: \"kubernetes.io/projected/70c7f356-3f7a-4b55-b032-4165e1864109-kube-api-access-xp8hg\") pod \"kuadrant-console-plugin-fbdd4f445-kl8dk\" (UID: \"70c7f356-3f7a-4b55-b032-4165e1864109\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kl8dk" Sep 11 14:47:27.503768 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:27.503677 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kl8dk" Sep 11 14:47:27.784796 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:27.784769 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-kl8dk"] Sep 11 14:47:27.786876 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:47:27.786844 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70c7f356_3f7a_4b55_b032_4165e1864109.slice/crio-7c268221bf41591cb90ea34c1a6e2bd8c2f29697feaa7aca35202f45f44b00de WatchSource:0}: Error finding container 7c268221bf41591cb90ea34c1a6e2bd8c2f29697feaa7aca35202f45f44b00de: Status 404 returned error can't find the container with id 7c268221bf41591cb90ea34c1a6e2bd8c2f29697feaa7aca35202f45f44b00de Sep 11 14:47:28.173209 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:28.173176 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pxbzq" Sep 11 14:47:28.228041 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:28.228006 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kl8dk" event={"ID":"70c7f356-3f7a-4b55-b032-4165e1864109","Type":"ContainerStarted","Data":"7c268221bf41591cb90ea34c1a6e2bd8c2f29697feaa7aca35202f45f44b00de"} Sep 11 14:47:33.256357 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:33.256317 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kl8dk" event={"ID":"70c7f356-3f7a-4b55-b032-4165e1864109","Type":"ContainerStarted","Data":"399d9dbf0fa62dc664beaff65d202bbdc36af3b4d6790b08a1a2bfcfc39bed4a"} Sep 11 14:47:33.275523 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:33.275477 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kl8dk" podStartSLOduration=1.832788052 podStartE2EDuration="6.275462527s" totalImagesPullingTime="4.442674475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:47:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:47:33.274551103 +0000 UTC m=+444.518536763" watchObservedRunningTime="2026-09-11 14:47:33.275462527 +0000 UTC m=+444.519448249" Sep 11 14:47:36.214771 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:36.214738 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rnh64" Sep 11 14:47:38.502676 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:38.502641 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rnh64"] Sep 11 14:47:38.503123 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:38.502858 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rnh64" podUID="b97b33a5-27aa-4b2d-92e6-9878b4b6915a" containerName="manager" containerID="cri-o://62344cf9a0565cb2f0771c63930f5291a09a206e2dd99e8f2346b96ec164fab0" gracePeriod=2 Sep 11 14:47:38.511127 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:38.511099 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rnh64"] Sep 11 14:47:38.531761 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:38.531735 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-gddvn"] Sep 11 14:47:38.532054 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:38.532040 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b97b33a5-27aa-4b2d-92e6-9878b4b6915a" containerName="manager" Sep 11 14:47:38.532108 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:38.532056 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b97b33a5-27aa-4b2d-92e6-9878b4b6915a" containerName="manager" Sep 11 14:47:38.532161 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:38.532136 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b97b33a5-27aa-4b2d-92e6-9878b4b6915a" containerName="manager" Sep 11 14:47:38.535437 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:38.535416 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gddvn" Sep 11 14:47:38.537865 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:47:38.537730 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rnh64\" is forbidden: User \"system:node:ip-10-0-135-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-39.ec2.internal' and this object" podUID="b97b33a5-27aa-4b2d-92e6-9878b4b6915a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rnh64" Sep 11 14:47:38.550948 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:38.550921 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-gddvn"] Sep 11 14:47:38.599101 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:38.599072 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p5rj\" (UniqueName: \"kubernetes.io/projected/3d239b94-d5e2-47ee-9800-285917fcc7f4-kube-api-access-5p5rj\") pod \"limitador-operator-controller-manager-74495c69f8-gddvn\" (UID: \"3d239b94-d5e2-47ee-9800-285917fcc7f4\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gddvn" Sep 11 14:47:38.700249 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:38.700219 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p5rj\" (UniqueName: \"kubernetes.io/projected/3d239b94-d5e2-47ee-9800-285917fcc7f4-kube-api-access-5p5rj\") pod \"limitador-operator-controller-manager-74495c69f8-gddvn\" (UID: \"3d239b94-d5e2-47ee-9800-285917fcc7f4\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gddvn" Sep 11 14:47:38.714181 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:38.714151 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p5rj\" (UniqueName: \"kubernetes.io/projected/3d239b94-d5e2-47ee-9800-285917fcc7f4-kube-api-access-5p5rj\") pod \"limitador-operator-controller-manager-74495c69f8-gddvn\" (UID: \"3d239b94-d5e2-47ee-9800-285917fcc7f4\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gddvn" Sep 11 14:47:38.734444 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:38.734420 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rnh64" Sep 11 14:47:38.736523 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:47:38.736500 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rnh64\" is forbidden: User \"system:node:ip-10-0-135-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-39.ec2.internal' and this object" podUID="b97b33a5-27aa-4b2d-92e6-9878b4b6915a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rnh64" Sep 11 14:47:38.801024 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:38.800973 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpjv6\" (UniqueName: \"kubernetes.io/projected/b97b33a5-27aa-4b2d-92e6-9878b4b6915a-kube-api-access-cpjv6\") pod \"b97b33a5-27aa-4b2d-92e6-9878b4b6915a\" (UID: \"b97b33a5-27aa-4b2d-92e6-9878b4b6915a\") " Sep 11 14:47:38.802858 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:38.802836 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97b33a5-27aa-4b2d-92e6-9878b4b6915a-kube-api-access-cpjv6" (OuterVolumeSpecName: "kube-api-access-cpjv6") pod "b97b33a5-27aa-4b2d-92e6-9878b4b6915a" (UID: "b97b33a5-27aa-4b2d-92e6-9878b4b6915a"). InnerVolumeSpecName "kube-api-access-cpjv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:47:38.877861 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:38.877838 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gddvn" Sep 11 14:47:38.902464 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:38.902440 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpjv6\" (UniqueName: \"kubernetes.io/projected/b97b33a5-27aa-4b2d-92e6-9878b4b6915a-kube-api-access-cpjv6\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:47:39.156888 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:39.156861 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-gddvn"] Sep 11 14:47:39.158898 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:47:39.158861 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d239b94_d5e2_47ee_9800_285917fcc7f4.slice/crio-611435bc0ebf88af1aa8ac8151331b562cb845f9bc94f2d53a7212a833271acb WatchSource:0}: Error finding container 611435bc0ebf88af1aa8ac8151331b562cb845f9bc94f2d53a7212a833271acb: Status 404 returned error can't find the container with id 611435bc0ebf88af1aa8ac8151331b562cb845f9bc94f2d53a7212a833271acb Sep 11 14:47:39.285045 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:39.285014 2583 generic.go:356] "Generic (PLEG): container finished" podID="b97b33a5-27aa-4b2d-92e6-9878b4b6915a" containerID="62344cf9a0565cb2f0771c63930f5291a09a206e2dd99e8f2346b96ec164fab0" exitCode=0 Sep 11 14:47:39.285176 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:39.285058 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rnh64" Sep 11 14:47:39.285176 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:39.285083 2583 scope.go:117] "RemoveContainer" containerID="62344cf9a0565cb2f0771c63930f5291a09a206e2dd99e8f2346b96ec164fab0" Sep 11 14:47:39.287016 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:39.286990 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gddvn" event={"ID":"3d239b94-d5e2-47ee-9800-285917fcc7f4","Type":"ContainerStarted","Data":"7e8357e18341a3bc3179bd93a49454cd32175b5522cd6776a5d2438733b75fa3"} Sep 11 14:47:39.287121 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:39.287029 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gddvn" event={"ID":"3d239b94-d5e2-47ee-9800-285917fcc7f4","Type":"ContainerStarted","Data":"611435bc0ebf88af1aa8ac8151331b562cb845f9bc94f2d53a7212a833271acb"} Sep 11 14:47:39.287121 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:39.287108 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gddvn" Sep 11 14:47:39.288307 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:47:39.288286 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rnh64\" is forbidden: User \"system:node:ip-10-0-135-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-39.ec2.internal' and this object" podUID="b97b33a5-27aa-4b2d-92e6-9878b4b6915a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rnh64" Sep 11 14:47:39.293416 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:39.293400 2583 scope.go:117] "RemoveContainer" containerID="62344cf9a0565cb2f0771c63930f5291a09a206e2dd99e8f2346b96ec164fab0" Sep 11 14:47:39.293696 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:47:39.293671 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62344cf9a0565cb2f0771c63930f5291a09a206e2dd99e8f2346b96ec164fab0\": container with ID starting with 62344cf9a0565cb2f0771c63930f5291a09a206e2dd99e8f2346b96ec164fab0 not found: ID does not exist" containerID="62344cf9a0565cb2f0771c63930f5291a09a206e2dd99e8f2346b96ec164fab0" Sep 11 14:47:39.293778 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:39.293727 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62344cf9a0565cb2f0771c63930f5291a09a206e2dd99e8f2346b96ec164fab0"} err="failed to get container status \"62344cf9a0565cb2f0771c63930f5291a09a206e2dd99e8f2346b96ec164fab0\": rpc error: code = NotFound desc = could not find container \"62344cf9a0565cb2f0771c63930f5291a09a206e2dd99e8f2346b96ec164fab0\": container with ID starting with 62344cf9a0565cb2f0771c63930f5291a09a206e2dd99e8f2346b96ec164fab0 not found: ID does not exist" Sep 11 14:47:39.321052 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:47:39.321022 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rnh64\" is forbidden: User \"system:node:ip-10-0-135-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-39.ec2.internal' and this object" podUID="b97b33a5-27aa-4b2d-92e6-9878b4b6915a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rnh64" Sep 11 14:47:39.323569 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:47:39.323547 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rnh64\" is forbidden: User \"system:node:ip-10-0-135-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-39.ec2.internal' and this object" podUID="b97b33a5-27aa-4b2d-92e6-9878b4b6915a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rnh64" Sep 11 14:47:39.329424 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:47:39.329392 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rnh64\" is forbidden: User \"system:node:ip-10-0-135-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-39.ec2.internal' and this object" podUID="b97b33a5-27aa-4b2d-92e6-9878b4b6915a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rnh64" Sep 11 14:47:39.331449 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:39.331426 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b97b33a5-27aa-4b2d-92e6-9878b4b6915a" path="/var/lib/kubelet/pods/b97b33a5-27aa-4b2d-92e6-9878b4b6915a/volumes" Sep 11 14:47:50.295419 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:50.295372 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gddvn" Sep 11 14:47:50.316208 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:47:50.316153 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gddvn" podStartSLOduration=12.316137272 podStartE2EDuration="12.316137272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:47:39.318690916 +0000 UTC m=+450.562676573" watchObservedRunningTime="2026-09-11 14:47:50.316137272 +0000 UTC m=+461.560122929" Sep 11 14:48:10.270946 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.270915 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn"] Sep 11 14:48:10.291525 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.291494 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.293123 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.293095 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn"] Sep 11 14:48:10.294904 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.294881 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-rxrrq\"" Sep 11 14:48:10.360080 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.360049 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6775488-0433-4467-a80a-db83b0ca7de9-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rp6sn\" (UID: \"b6775488-0433-4467-a80a-db83b0ca7de9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.360080 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.360083 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6775488-0433-4467-a80a-db83b0ca7de9-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rp6sn\" (UID: \"b6775488-0433-4467-a80a-db83b0ca7de9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.360292 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.360111 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6775488-0433-4467-a80a-db83b0ca7de9-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rp6sn\" (UID: \"b6775488-0433-4467-a80a-db83b0ca7de9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.360292 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.360160 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6775488-0433-4467-a80a-db83b0ca7de9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rp6sn\" (UID: \"b6775488-0433-4467-a80a-db83b0ca7de9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.360292 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.360204 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99vh9\" (UniqueName: \"kubernetes.io/projected/b6775488-0433-4467-a80a-db83b0ca7de9-kube-api-access-99vh9\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rp6sn\" (UID: \"b6775488-0433-4467-a80a-db83b0ca7de9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.360292 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.360232 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6775488-0433-4467-a80a-db83b0ca7de9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rp6sn\" (UID: \"b6775488-0433-4467-a80a-db83b0ca7de9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.360292 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.360255 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6775488-0433-4467-a80a-db83b0ca7de9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rp6sn\" (UID: \"b6775488-0433-4467-a80a-db83b0ca7de9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.360292 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.360275 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6775488-0433-4467-a80a-db83b0ca7de9-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rp6sn\" (UID: \"b6775488-0433-4467-a80a-db83b0ca7de9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.360481 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.360299 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6775488-0433-4467-a80a-db83b0ca7de9-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rp6sn\" (UID: \"b6775488-0433-4467-a80a-db83b0ca7de9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.461482 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.461449 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6775488-0433-4467-a80a-db83b0ca7de9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rp6sn\" (UID: \"b6775488-0433-4467-a80a-db83b0ca7de9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.461482 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.461488 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6775488-0433-4467-a80a-db83b0ca7de9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rp6sn\" (UID: \"b6775488-0433-4467-a80a-db83b0ca7de9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.461759 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.461508 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6775488-0433-4467-a80a-db83b0ca7de9-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rp6sn\" (UID: \"b6775488-0433-4467-a80a-db83b0ca7de9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.461759 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.461536 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6775488-0433-4467-a80a-db83b0ca7de9-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rp6sn\" (UID: \"b6775488-0433-4467-a80a-db83b0ca7de9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.461759 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.461577 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6775488-0433-4467-a80a-db83b0ca7de9-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rp6sn\" (UID: \"b6775488-0433-4467-a80a-db83b0ca7de9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.461759 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.461603 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6775488-0433-4467-a80a-db83b0ca7de9-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rp6sn\" (UID: \"b6775488-0433-4467-a80a-db83b0ca7de9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.461759 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.461645 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6775488-0433-4467-a80a-db83b0ca7de9-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rp6sn\" (UID: \"b6775488-0433-4467-a80a-db83b0ca7de9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.461759 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.461673 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6775488-0433-4467-a80a-db83b0ca7de9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rp6sn\" (UID: \"b6775488-0433-4467-a80a-db83b0ca7de9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.461759 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.461737 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99vh9\" (UniqueName: \"kubernetes.io/projected/b6775488-0433-4467-a80a-db83b0ca7de9-kube-api-access-99vh9\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rp6sn\" (UID: \"b6775488-0433-4467-a80a-db83b0ca7de9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.462094 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.461965 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6775488-0433-4467-a80a-db83b0ca7de9-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rp6sn\" (UID: \"b6775488-0433-4467-a80a-db83b0ca7de9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.462145 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.462117 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6775488-0433-4467-a80a-db83b0ca7de9-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rp6sn\" (UID: \"b6775488-0433-4467-a80a-db83b0ca7de9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.462195 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.462179 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6775488-0433-4467-a80a-db83b0ca7de9-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rp6sn\" (UID: \"b6775488-0433-4467-a80a-db83b0ca7de9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.462254 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.462235 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6775488-0433-4467-a80a-db83b0ca7de9-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rp6sn\" (UID: \"b6775488-0433-4467-a80a-db83b0ca7de9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.462309 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.462290 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6775488-0433-4467-a80a-db83b0ca7de9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rp6sn\" (UID: \"b6775488-0433-4467-a80a-db83b0ca7de9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.463983 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.463962 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6775488-0433-4467-a80a-db83b0ca7de9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rp6sn\" (UID: \"b6775488-0433-4467-a80a-db83b0ca7de9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.464198 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.464178 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6775488-0433-4467-a80a-db83b0ca7de9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rp6sn\" (UID: \"b6775488-0433-4467-a80a-db83b0ca7de9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.474503 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.474457 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6775488-0433-4467-a80a-db83b0ca7de9-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rp6sn\" (UID: \"b6775488-0433-4467-a80a-db83b0ca7de9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.474590 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.474507 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99vh9\" (UniqueName: \"kubernetes.io/projected/b6775488-0433-4467-a80a-db83b0ca7de9-kube-api-access-99vh9\") pod \"maas-default-gateway-openshift-default-5ccb968c66-rp6sn\" (UID: \"b6775488-0433-4467-a80a-db83b0ca7de9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.601973 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.601898 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:10.877030 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.876999 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn"] Sep 11 14:48:10.880373 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:48:10.880342 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6775488_0433_4467_a80a_db83b0ca7de9.slice/crio-fc626a533b00c629ba3b379a23fd84f6ae816be3e37712bd2095ba12057a88d3 WatchSource:0}: Error finding container fc626a533b00c629ba3b379a23fd84f6ae816be3e37712bd2095ba12057a88d3: Status 404 returned error can't find the container with id fc626a533b00c629ba3b379a23fd84f6ae816be3e37712bd2095ba12057a88d3 Sep 11 14:48:10.882632 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.882598 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:48:10.882722 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.882663 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:48:10.882722 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:10.882690 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:48:11.447440 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:11.447404 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" event={"ID":"b6775488-0433-4467-a80a-db83b0ca7de9","Type":"ContainerStarted","Data":"f9b184cc12541f72bb566f237cdacf73c92e3bb48cd328634b3b763d0da5ec68"} Sep 11 14:48:11.447929 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:11.447464 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" event={"ID":"b6775488-0433-4467-a80a-db83b0ca7de9","Type":"ContainerStarted","Data":"fc626a533b00c629ba3b379a23fd84f6ae816be3e37712bd2095ba12057a88d3"} Sep 11 14:48:11.472875 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:11.472830 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" podStartSLOduration=1.472816299 podStartE2EDuration="1.472816299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:48:11.471841687 +0000 UTC m=+482.715827357" watchObservedRunningTime="2026-09-11 14:48:11.472816299 +0000 UTC m=+482.716801956" Sep 11 14:48:11.602509 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:11.602456 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:11.603983 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:11.603958 2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-rp6sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 11 14:48:11.604097 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:11.604004 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" podUID="b6775488-0433-4467-a80a-db83b0ca7de9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 11 14:48:12.606278 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:12.606241 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:13.457440 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:13.457402 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:13.458302 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:13.458283 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:48:27.114878 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:27.114838 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-c4hcw"] Sep 11 14:48:27.118243 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:27.118222 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-c4hcw" Sep 11 14:48:27.120808 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:27.120790 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vmn8r\"" Sep 11 14:48:27.128320 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:27.128294 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-c4hcw"] Sep 11 14:48:27.200057 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:27.200020 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kb7t\" (UniqueName: \"kubernetes.io/projected/7fde779e-08d3-4c92-9868-a7c03341699d-kube-api-access-8kb7t\") pod \"authorino-7bcfd8f9cd-c4hcw\" (UID: \"7fde779e-08d3-4c92-9868-a7c03341699d\") " pod="kuadrant-system/authorino-7bcfd8f9cd-c4hcw" Sep 11 14:48:27.301474 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:27.301419 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kb7t\" (UniqueName: \"kubernetes.io/projected/7fde779e-08d3-4c92-9868-a7c03341699d-kube-api-access-8kb7t\") pod \"authorino-7bcfd8f9cd-c4hcw\" (UID: \"7fde779e-08d3-4c92-9868-a7c03341699d\") " pod="kuadrant-system/authorino-7bcfd8f9cd-c4hcw" Sep 11 14:48:27.313579 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:27.313545 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kb7t\" (UniqueName: \"kubernetes.io/projected/7fde779e-08d3-4c92-9868-a7c03341699d-kube-api-access-8kb7t\") pod \"authorino-7bcfd8f9cd-c4hcw\" (UID: \"7fde779e-08d3-4c92-9868-a7c03341699d\") " pod="kuadrant-system/authorino-7bcfd8f9cd-c4hcw" Sep 11 14:48:27.427998 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:27.427964 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-c4hcw" Sep 11 14:48:27.700764 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:27.700675 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-c4hcw"] Sep 11 14:48:27.704009 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:48:27.703985 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fde779e_08d3_4c92_9868_a7c03341699d.slice/crio-24ad87d3877c584645c7259ff74a56ddc6eef8289014fef4c940245802d024c6 WatchSource:0}: Error finding container 24ad87d3877c584645c7259ff74a56ddc6eef8289014fef4c940245802d024c6: Status 404 returned error can't find the container with id 24ad87d3877c584645c7259ff74a56ddc6eef8289014fef4c940245802d024c6 Sep 11 14:48:28.530257 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:28.530219 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-c4hcw" event={"ID":"7fde779e-08d3-4c92-9868-a7c03341699d","Type":"ContainerStarted","Data":"24ad87d3877c584645c7259ff74a56ddc6eef8289014fef4c940245802d024c6"} Sep 11 14:48:30.543952 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:30.543909 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-c4hcw" event={"ID":"7fde779e-08d3-4c92-9868-a7c03341699d","Type":"ContainerStarted","Data":"c2ad05237a1eb91e7ac2e3ca177d390e333d2a620ab9ac20f00fe6f39bce277b"} Sep 11 14:48:30.561693 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:30.561638 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-c4hcw" podStartSLOduration=1.63273449 podStartE2EDuration="3.56161977s" totalImagesPullingTime="1.92888528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:48:30.560150704 +0000 UTC m=+501.804136365" watchObservedRunningTime="2026-09-11 14:48:30.56161977 +0000 UTC m=+501.805605426" Sep 11 14:48:30.591468 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:30.591432 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-c4hcw"] Sep 11 14:48:32.554285 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:32.554246 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-c4hcw" podUID="7fde779e-08d3-4c92-9868-a7c03341699d" containerName="authorino" containerID="cri-o://c2ad05237a1eb91e7ac2e3ca177d390e333d2a620ab9ac20f00fe6f39bce277b" gracePeriod=30 Sep 11 14:48:32.798776 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:32.798751 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-c4hcw" Sep 11 14:48:32.951329 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:32.951289 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kb7t\" (UniqueName: \"kubernetes.io/projected/7fde779e-08d3-4c92-9868-a7c03341699d-kube-api-access-8kb7t\") pod \"7fde779e-08d3-4c92-9868-a7c03341699d\" (UID: \"7fde779e-08d3-4c92-9868-a7c03341699d\") " Sep 11 14:48:32.953414 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:32.953381 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fde779e-08d3-4c92-9868-a7c03341699d-kube-api-access-8kb7t" (OuterVolumeSpecName: "kube-api-access-8kb7t") pod "7fde779e-08d3-4c92-9868-a7c03341699d" (UID: "7fde779e-08d3-4c92-9868-a7c03341699d"). InnerVolumeSpecName "kube-api-access-8kb7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:48:33.052499 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:33.052460 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kb7t\" (UniqueName: \"kubernetes.io/projected/7fde779e-08d3-4c92-9868-a7c03341699d-kube-api-access-8kb7t\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:48:33.566388 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:33.566352 2583 generic.go:356] "Generic (PLEG): container finished" podID="7fde779e-08d3-4c92-9868-a7c03341699d" containerID="c2ad05237a1eb91e7ac2e3ca177d390e333d2a620ab9ac20f00fe6f39bce277b" exitCode=0 Sep 11 14:48:33.566862 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:33.566399 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-c4hcw" Sep 11 14:48:33.566862 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:33.566441 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-c4hcw" event={"ID":"7fde779e-08d3-4c92-9868-a7c03341699d","Type":"ContainerDied","Data":"c2ad05237a1eb91e7ac2e3ca177d390e333d2a620ab9ac20f00fe6f39bce277b"} Sep 11 14:48:33.566862 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:33.566500 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-c4hcw" event={"ID":"7fde779e-08d3-4c92-9868-a7c03341699d","Type":"ContainerDied","Data":"24ad87d3877c584645c7259ff74a56ddc6eef8289014fef4c940245802d024c6"} Sep 11 14:48:33.566862 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:33.566517 2583 scope.go:117] "RemoveContainer" containerID="c2ad05237a1eb91e7ac2e3ca177d390e333d2a620ab9ac20f00fe6f39bce277b" Sep 11 14:48:33.577415 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:33.577391 2583 scope.go:117] "RemoveContainer" containerID="c2ad05237a1eb91e7ac2e3ca177d390e333d2a620ab9ac20f00fe6f39bce277b" Sep 11 14:48:33.577889 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:48:33.577861 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2ad05237a1eb91e7ac2e3ca177d390e333d2a620ab9ac20f00fe6f39bce277b\": container with ID starting with c2ad05237a1eb91e7ac2e3ca177d390e333d2a620ab9ac20f00fe6f39bce277b not found: ID does not exist" containerID="c2ad05237a1eb91e7ac2e3ca177d390e333d2a620ab9ac20f00fe6f39bce277b" Sep 11 14:48:33.577986 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:33.577892 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2ad05237a1eb91e7ac2e3ca177d390e333d2a620ab9ac20f00fe6f39bce277b"} err="failed to get container status \"c2ad05237a1eb91e7ac2e3ca177d390e333d2a620ab9ac20f00fe6f39bce277b\": rpc error: code = NotFound desc = could not find container \"c2ad05237a1eb91e7ac2e3ca177d390e333d2a620ab9ac20f00fe6f39bce277b\": container with ID starting with c2ad05237a1eb91e7ac2e3ca177d390e333d2a620ab9ac20f00fe6f39bce277b not found: ID does not exist" Sep 11 14:48:33.585714 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:33.585679 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-c4hcw"] Sep 11 14:48:33.589083 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:33.589033 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-c4hcw"] Sep 11 14:48:35.329572 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:35.329542 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fde779e-08d3-4c92-9868-a7c03341699d" path="/var/lib/kubelet/pods/7fde779e-08d3-4c92-9868-a7c03341699d/volumes" Sep 11 14:48:58.766234 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:58.766202 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-shnkm"] Sep 11 14:48:58.766806 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:58.766678 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fde779e-08d3-4c92-9868-a7c03341699d" containerName="authorino" Sep 11 14:48:58.766806 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:58.766698 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fde779e-08d3-4c92-9868-a7c03341699d" containerName="authorino" Sep 11 14:48:58.766938 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:58.766818 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fde779e-08d3-4c92-9868-a7c03341699d" containerName="authorino" Sep 11 14:48:58.771113 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:58.771094 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-shnkm" Sep 11 14:48:58.774075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:58.774054 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 11 14:48:58.774188 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:58.774057 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vmn8r\"" Sep 11 14:48:58.777547 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:58.777522 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-shnkm"] Sep 11 14:48:58.862541 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:58.862510 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mzf2\" (UniqueName: \"kubernetes.io/projected/950a06fd-95ca-4f49-83df-b773bdfeace8-kube-api-access-2mzf2\") pod \"authorino-6d4dfc649f-shnkm\" (UID: \"950a06fd-95ca-4f49-83df-b773bdfeace8\") " pod="kuadrant-system/authorino-6d4dfc649f-shnkm" Sep 11 14:48:58.862655 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:58.862550 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/950a06fd-95ca-4f49-83df-b773bdfeace8-tls-cert\") pod \"authorino-6d4dfc649f-shnkm\" (UID: \"950a06fd-95ca-4f49-83df-b773bdfeace8\") " pod="kuadrant-system/authorino-6d4dfc649f-shnkm" Sep 11 14:48:58.949380 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:58.949356 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-shnkm"] Sep 11 14:48:58.949638 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:48:58.949616 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-2mzf2 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-shnkm" podUID="950a06fd-95ca-4f49-83df-b773bdfeace8" Sep 11 14:48:58.963095 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:58.963073 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mzf2\" (UniqueName: \"kubernetes.io/projected/950a06fd-95ca-4f49-83df-b773bdfeace8-kube-api-access-2mzf2\") pod \"authorino-6d4dfc649f-shnkm\" (UID: \"950a06fd-95ca-4f49-83df-b773bdfeace8\") " pod="kuadrant-system/authorino-6d4dfc649f-shnkm" Sep 11 14:48:58.963177 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:58.963109 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/950a06fd-95ca-4f49-83df-b773bdfeace8-tls-cert\") pod \"authorino-6d4dfc649f-shnkm\" (UID: \"950a06fd-95ca-4f49-83df-b773bdfeace8\") " pod="kuadrant-system/authorino-6d4dfc649f-shnkm" Sep 11 14:48:58.965416 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:58.965397 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/950a06fd-95ca-4f49-83df-b773bdfeace8-tls-cert\") pod \"authorino-6d4dfc649f-shnkm\" (UID: \"950a06fd-95ca-4f49-83df-b773bdfeace8\") " pod="kuadrant-system/authorino-6d4dfc649f-shnkm" Sep 11 14:48:58.973362 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:58.973337 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mzf2\" (UniqueName: \"kubernetes.io/projected/950a06fd-95ca-4f49-83df-b773bdfeace8-kube-api-access-2mzf2\") pod \"authorino-6d4dfc649f-shnkm\" (UID: \"950a06fd-95ca-4f49-83df-b773bdfeace8\") " pod="kuadrant-system/authorino-6d4dfc649f-shnkm" Sep 11 14:48:58.977378 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:58.977358 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65c5c88497-kbb4p"] Sep 11 14:48:58.980693 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:58.980676 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65c5c88497-kbb4p" Sep 11 14:48:58.994060 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:58.994039 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65c5c88497-kbb4p"] Sep 11 14:48:59.064135 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:59.064064 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f125bd03-314d-4987-a471-bdbe74de3feb-tls-cert\") pod \"authorino-65c5c88497-kbb4p\" (UID: \"f125bd03-314d-4987-a471-bdbe74de3feb\") " pod="kuadrant-system/authorino-65c5c88497-kbb4p" Sep 11 14:48:59.064135 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:59.064106 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns2tq\" (UniqueName: \"kubernetes.io/projected/f125bd03-314d-4987-a471-bdbe74de3feb-kube-api-access-ns2tq\") pod \"authorino-65c5c88497-kbb4p\" (UID: \"f125bd03-314d-4987-a471-bdbe74de3feb\") " pod="kuadrant-system/authorino-65c5c88497-kbb4p" Sep 11 14:48:59.165351 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:59.165326 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f125bd03-314d-4987-a471-bdbe74de3feb-tls-cert\") pod \"authorino-65c5c88497-kbb4p\" (UID: \"f125bd03-314d-4987-a471-bdbe74de3feb\") " pod="kuadrant-system/authorino-65c5c88497-kbb4p" Sep 11 14:48:59.165465 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:59.165360 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns2tq\" (UniqueName: \"kubernetes.io/projected/f125bd03-314d-4987-a471-bdbe74de3feb-kube-api-access-ns2tq\") pod \"authorino-65c5c88497-kbb4p\" (UID: \"f125bd03-314d-4987-a471-bdbe74de3feb\") " pod="kuadrant-system/authorino-65c5c88497-kbb4p" Sep 11 14:48:59.167819 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:59.167802 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f125bd03-314d-4987-a471-bdbe74de3feb-tls-cert\") pod \"authorino-65c5c88497-kbb4p\" (UID: \"f125bd03-314d-4987-a471-bdbe74de3feb\") " pod="kuadrant-system/authorino-65c5c88497-kbb4p" Sep 11 14:48:59.175752 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:59.175731 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns2tq\" (UniqueName: \"kubernetes.io/projected/f125bd03-314d-4987-a471-bdbe74de3feb-kube-api-access-ns2tq\") pod \"authorino-65c5c88497-kbb4p\" (UID: \"f125bd03-314d-4987-a471-bdbe74de3feb\") " pod="kuadrant-system/authorino-65c5c88497-kbb4p" Sep 11 14:48:59.297086 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:59.297068 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65c5c88497-kbb4p" Sep 11 14:48:59.567468 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:59.567440 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65c5c88497-kbb4p"] Sep 11 14:48:59.570056 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:48:59.570027 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf125bd03_314d_4987_a471_bdbe74de3feb.slice/crio-dcb75c56403414c8be425ac47688d7db9d038fe987993de3b20853b2567d9ece WatchSource:0}: Error finding container dcb75c56403414c8be425ac47688d7db9d038fe987993de3b20853b2567d9ece: Status 404 returned error can't find the container with id dcb75c56403414c8be425ac47688d7db9d038fe987993de3b20853b2567d9ece Sep 11 14:48:59.696195 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:59.696171 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-shnkm" Sep 11 14:48:59.696330 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:59.696163 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65c5c88497-kbb4p" event={"ID":"f125bd03-314d-4987-a471-bdbe74de3feb","Type":"ContainerStarted","Data":"dcb75c56403414c8be425ac47688d7db9d038fe987993de3b20853b2567d9ece"} Sep 11 14:48:59.704307 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:59.704290 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-shnkm" Sep 11 14:48:59.768876 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:59.768857 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mzf2\" (UniqueName: \"kubernetes.io/projected/950a06fd-95ca-4f49-83df-b773bdfeace8-kube-api-access-2mzf2\") pod \"950a06fd-95ca-4f49-83df-b773bdfeace8\" (UID: \"950a06fd-95ca-4f49-83df-b773bdfeace8\") " Sep 11 14:48:59.769251 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:59.768908 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/950a06fd-95ca-4f49-83df-b773bdfeace8-tls-cert\") pod \"950a06fd-95ca-4f49-83df-b773bdfeace8\" (UID: \"950a06fd-95ca-4f49-83df-b773bdfeace8\") " Sep 11 14:48:59.770947 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:59.770927 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/950a06fd-95ca-4f49-83df-b773bdfeace8-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "950a06fd-95ca-4f49-83df-b773bdfeace8" (UID: "950a06fd-95ca-4f49-83df-b773bdfeace8"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:48:59.771016 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:59.770938 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950a06fd-95ca-4f49-83df-b773bdfeace8-kube-api-access-2mzf2" (OuterVolumeSpecName: "kube-api-access-2mzf2") pod "950a06fd-95ca-4f49-83df-b773bdfeace8" (UID: "950a06fd-95ca-4f49-83df-b773bdfeace8"). InnerVolumeSpecName "kube-api-access-2mzf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:48:59.869696 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:59.869675 2583 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/950a06fd-95ca-4f49-83df-b773bdfeace8-tls-cert\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:48:59.869696 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:48:59.869693 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mzf2\" (UniqueName: \"kubernetes.io/projected/950a06fd-95ca-4f49-83df-b773bdfeace8-kube-api-access-2mzf2\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:49:00.703335 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:00.703298 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65c5c88497-kbb4p" event={"ID":"f125bd03-314d-4987-a471-bdbe74de3feb","Type":"ContainerStarted","Data":"0b2d8a9dc5e6d5d4bbab924d186672ad292a897eff657f2d5a0afa55dab22790"} Sep 11 14:49:00.703335 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:00.703332 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-shnkm" Sep 11 14:49:00.719373 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:00.719331 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-65c5c88497-kbb4p" podStartSLOduration=2.355396047 podStartE2EDuration="2.719316473s" totalImagesPullingTime="363.920426ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:49:00.718615496 +0000 UTC m=+531.962601157" watchObservedRunningTime="2026-09-11 14:49:00.719316473 +0000 UTC m=+531.963302131" Sep 11 14:49:00.748163 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:00.748130 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-shnkm"] Sep 11 14:49:00.756698 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:00.756673 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-shnkm"] Sep 11 14:49:01.330063 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:01.330031 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="950a06fd-95ca-4f49-83df-b773bdfeace8" path="/var/lib/kubelet/pods/950a06fd-95ca-4f49-83df-b773bdfeace8/volumes" Sep 11 14:49:45.430353 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:45.430321 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-xc4hx"] Sep 11 14:49:45.433680 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:45.433660 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xc4hx" Sep 11 14:49:45.437093 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:45.437072 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 11 14:49:45.438104 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:45.438084 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-4xm7h\"" Sep 11 14:49:45.447619 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:45.447585 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-xc4hx"] Sep 11 14:49:45.521491 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:45.521450 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dz9s\" (UniqueName: \"kubernetes.io/projected/b5e55d5d-7bdf-4a23-af47-fefd40e063d4-kube-api-access-4dz9s\") pod \"payload-pre-processing-59fbc47bf8-xc4hx\" (UID: \"b5e55d5d-7bdf-4a23-af47-fefd40e063d4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xc4hx" Sep 11 14:49:45.521656 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:45.521509 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5e55d5d-7bdf-4a23-af47-fefd40e063d4-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-xc4hx\" (UID: \"b5e55d5d-7bdf-4a23-af47-fefd40e063d4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xc4hx" Sep 11 14:49:45.622382 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:45.622330 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5e55d5d-7bdf-4a23-af47-fefd40e063d4-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-xc4hx\" (UID: \"b5e55d5d-7bdf-4a23-af47-fefd40e063d4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xc4hx" Sep 11 14:49:45.622566 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:45.622429 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dz9s\" (UniqueName: \"kubernetes.io/projected/b5e55d5d-7bdf-4a23-af47-fefd40e063d4-kube-api-access-4dz9s\") pod \"payload-pre-processing-59fbc47bf8-xc4hx\" (UID: \"b5e55d5d-7bdf-4a23-af47-fefd40e063d4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xc4hx" Sep 11 14:49:45.623348 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:45.623324 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5e55d5d-7bdf-4a23-af47-fefd40e063d4-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-xc4hx\" (UID: \"b5e55d5d-7bdf-4a23-af47-fefd40e063d4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xc4hx" Sep 11 14:49:45.633931 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:45.633905 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dz9s\" (UniqueName: \"kubernetes.io/projected/b5e55d5d-7bdf-4a23-af47-fefd40e063d4-kube-api-access-4dz9s\") pod \"payload-pre-processing-59fbc47bf8-xc4hx\" (UID: \"b5e55d5d-7bdf-4a23-af47-fefd40e063d4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xc4hx" Sep 11 14:49:45.743807 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:45.743684 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xc4hx" Sep 11 14:49:46.021127 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:46.020936 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-xc4hx"] Sep 11 14:49:46.023307 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:49:46.023279 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e55d5d_7bdf_4a23_af47_fefd40e063d4.slice/crio-0c30221ecda72844bbe7957fc8df0377c936c6ce10abee0d7bab4dd81becb05e WatchSource:0}: Error finding container 0c30221ecda72844bbe7957fc8df0377c936c6ce10abee0d7bab4dd81becb05e: Status 404 returned error can't find the container with id 0c30221ecda72844bbe7957fc8df0377c936c6ce10abee0d7bab4dd81becb05e Sep 11 14:49:46.934587 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:46.934514 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xc4hx" event={"ID":"b5e55d5d-7bdf-4a23-af47-fefd40e063d4","Type":"ContainerStarted","Data":"0c30221ecda72844bbe7957fc8df0377c936c6ce10abee0d7bab4dd81becb05e"} Sep 11 14:49:48.959425 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:48.959329 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xc4hx" event={"ID":"b5e55d5d-7bdf-4a23-af47-fefd40e063d4","Type":"ContainerStarted","Data":"185898bd045cfd1fb09c0df621f2ab0998a098480ee3f718a9913c396ef48ceb"} Sep 11 14:49:48.959837 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:48.959429 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xc4hx" Sep 11 14:49:48.977282 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:48.977233 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xc4hx" podStartSLOduration=1.329327256 podStartE2EDuration="3.97721781s" totalImagesPullingTime="2.647890554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:49:48.976758811 +0000 UTC m=+580.220744470" watchObservedRunningTime="2026-09-11 14:49:48.97721781 +0000 UTC m=+580.221203468" Sep 11 14:49:54.992396 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:54.992363 2583 generic.go:356] "Generic (PLEG): container finished" podID="b6775488-0433-4467-a80a-db83b0ca7de9" containerID="f9b184cc12541f72bb566f237cdacf73c92e3bb48cd328634b3b763d0da5ec68" exitCode=0 Sep 11 14:49:54.992781 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:54.992435 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" event={"ID":"b6775488-0433-4467-a80a-db83b0ca7de9","Type":"ContainerDied","Data":"f9b184cc12541f72bb566f237cdacf73c92e3bb48cd328634b3b763d0da5ec68"} Sep 11 14:49:54.992823 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:54.992801 2583 scope.go:117] "RemoveContainer" containerID="f9b184cc12541f72bb566f237cdacf73c92e3bb48cd328634b3b763d0da5ec68" Sep 11 14:49:54.993688 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:54.993664 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:49:54.993758 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:54.993742 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:49:54.993796 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:54.993776 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:49:55.999932 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:55.999898 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" event={"ID":"b6775488-0433-4467-a80a-db83b0ca7de9","Type":"ContainerStarted","Data":"f41f008fc906e8650530765f4de407aa6842dfd32e785f9f40b8a2058e19951e"} Sep 11 14:49:56.602977 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:56.602939 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:49:56.603920 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:56.603892 2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-rp6sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 11 14:49:56.604057 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:56.603952 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" podUID="b6775488-0433-4467-a80a-db83b0ca7de9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 11 14:49:57.603026 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:57.602993 2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-rp6sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 11 14:49:57.603394 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:57.603043 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" podUID="b6775488-0433-4467-a80a-db83b0ca7de9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 11 14:49:58.458098 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:58.458049 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:49:58.602960 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:58.602929 2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-rp6sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 11 14:49:58.603101 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:58.602980 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" podUID="b6775488-0433-4467-a80a-db83b0ca7de9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 11 14:49:59.603232 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:59.603178 2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-rp6sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 11 14:49:59.603644 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:59.603261 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" podUID="b6775488-0433-4467-a80a-db83b0ca7de9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 11 14:49:59.968334 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:49:59.968300 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xc4hx" Sep 11 14:50:00.602954 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:00.602919 2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-rp6sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 11 14:50:00.603126 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:00.602983 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" podUID="b6775488-0433-4467-a80a-db83b0ca7de9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 11 14:50:01.606351 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:01.606318 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:50:01.607441 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:01.607420 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-rp6sn" Sep 11 14:50:09.169453 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:09.169422 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lwrt2_57d4ba26-384d-42f8-86a1-335e27cdf762/console-operator/1.log" Sep 11 14:50:09.170420 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:09.170397 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lwrt2_57d4ba26-384d-42f8-86a1-335e27cdf762/console-operator/1.log" Sep 11 14:50:55.675363 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:55.675319 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb"] Sep 11 14:50:55.680843 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:55.680818 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb" Sep 11 14:50:55.684345 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:55.684317 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 11 14:50:55.684345 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:55.684331 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 11 14:50:55.684527 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:55.684371 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-cp687\"" Sep 11 14:50:55.684527 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:55.684322 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 11 14:50:55.684644 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:55.684618 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb"] Sep 11 14:50:55.828949 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:55.828921 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/848c1889-60dc-4d88-b467-a1c51f9bf3b2-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb\" (UID: \"848c1889-60dc-4d88-b467-a1c51f9bf3b2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb" Sep 11 14:50:55.829182 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:55.828994 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/848c1889-60dc-4d88-b467-a1c51f9bf3b2-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb\" (UID: \"848c1889-60dc-4d88-b467-a1c51f9bf3b2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb" Sep 11 14:50:55.829182 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:55.829031 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/848c1889-60dc-4d88-b467-a1c51f9bf3b2-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb\" (UID: \"848c1889-60dc-4d88-b467-a1c51f9bf3b2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb" Sep 11 14:50:55.829182 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:55.829056 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvzv7\" (UniqueName: \"kubernetes.io/projected/848c1889-60dc-4d88-b467-a1c51f9bf3b2-kube-api-access-fvzv7\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb\" (UID: \"848c1889-60dc-4d88-b467-a1c51f9bf3b2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb" Sep 11 14:50:55.829182 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:55.829139 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/848c1889-60dc-4d88-b467-a1c51f9bf3b2-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb\" (UID: \"848c1889-60dc-4d88-b467-a1c51f9bf3b2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb" Sep 11 14:50:55.829182 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:55.829170 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/848c1889-60dc-4d88-b467-a1c51f9bf3b2-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb\" (UID: \"848c1889-60dc-4d88-b467-a1c51f9bf3b2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb" Sep 11 14:50:55.930160 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:55.930082 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/848c1889-60dc-4d88-b467-a1c51f9bf3b2-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb\" (UID: \"848c1889-60dc-4d88-b467-a1c51f9bf3b2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb" Sep 11 14:50:55.930160 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:55.930121 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/848c1889-60dc-4d88-b467-a1c51f9bf3b2-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb\" (UID: \"848c1889-60dc-4d88-b467-a1c51f9bf3b2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb" Sep 11 14:50:55.930160 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:55.930156 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/848c1889-60dc-4d88-b467-a1c51f9bf3b2-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb\" (UID: \"848c1889-60dc-4d88-b467-a1c51f9bf3b2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb" Sep 11 14:50:55.930413 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:55.930206 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/848c1889-60dc-4d88-b467-a1c51f9bf3b2-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb\" (UID: \"848c1889-60dc-4d88-b467-a1c51f9bf3b2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb" Sep 11 14:50:55.930413 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:55.930238 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/848c1889-60dc-4d88-b467-a1c51f9bf3b2-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb\" (UID: \"848c1889-60dc-4d88-b467-a1c51f9bf3b2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb" Sep 11 14:50:55.930413 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:55.930257 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvzv7\" (UniqueName: \"kubernetes.io/projected/848c1889-60dc-4d88-b467-a1c51f9bf3b2-kube-api-access-fvzv7\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb\" (UID: \"848c1889-60dc-4d88-b467-a1c51f9bf3b2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb" Sep 11 14:50:55.930576 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:55.930485 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/848c1889-60dc-4d88-b467-a1c51f9bf3b2-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb\" (UID: \"848c1889-60dc-4d88-b467-a1c51f9bf3b2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb" Sep 11 14:50:55.930576 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:55.930520 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/848c1889-60dc-4d88-b467-a1c51f9bf3b2-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb\" (UID: \"848c1889-60dc-4d88-b467-a1c51f9bf3b2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb" Sep 11 14:50:55.930663 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:55.930646 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/848c1889-60dc-4d88-b467-a1c51f9bf3b2-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb\" (UID: \"848c1889-60dc-4d88-b467-a1c51f9bf3b2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb" Sep 11 14:50:55.932385 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:55.932365 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/848c1889-60dc-4d88-b467-a1c51f9bf3b2-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb\" (UID: \"848c1889-60dc-4d88-b467-a1c51f9bf3b2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb" Sep 11 14:50:55.932667 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:55.932648 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/848c1889-60dc-4d88-b467-a1c51f9bf3b2-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb\" (UID: \"848c1889-60dc-4d88-b467-a1c51f9bf3b2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb" Sep 11 14:50:55.939450 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:55.939428 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvzv7\" (UniqueName: \"kubernetes.io/projected/848c1889-60dc-4d88-b467-a1c51f9bf3b2-kube-api-access-fvzv7\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb\" (UID: \"848c1889-60dc-4d88-b467-a1c51f9bf3b2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb" Sep 11 14:50:55.992800 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:55.992780 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb" Sep 11 14:50:56.265700 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:56.265674 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb"] Sep 11 14:50:56.266768 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:50:56.266742 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod848c1889_60dc_4d88_b467_a1c51f9bf3b2.slice/crio-c674a5c873830236a4f06c13aaa0e3dfe8b4b1c807e24557143d75a26ff85e5c WatchSource:0}: Error finding container c674a5c873830236a4f06c13aaa0e3dfe8b4b1c807e24557143d75a26ff85e5c: Status 404 returned error can't find the container with id c674a5c873830236a4f06c13aaa0e3dfe8b4b1c807e24557143d75a26ff85e5c Sep 11 14:50:56.268183 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:56.268164 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:50:56.299560 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:56.299531 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb" event={"ID":"848c1889-60dc-4d88-b467-a1c51f9bf3b2","Type":"ContainerStarted","Data":"c674a5c873830236a4f06c13aaa0e3dfe8b4b1c807e24557143d75a26ff85e5c"} Sep 11 14:50:58.311221 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:58.311183 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb" event={"ID":"848c1889-60dc-4d88-b467-a1c51f9bf3b2","Type":"ContainerStarted","Data":"58313ba77941d9823f1cb6a7cdee83011905bf760f446d59148b26d4f98e645f"} Sep 11 14:50:58.332887 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:50:58.332841 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb" podStartSLOduration=2.259008669 podStartE2EDuration="3.33282619s" totalImagesPullingTime="1.073817521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:50:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:50:58.330212248 +0000 UTC m=+649.574197906" watchObservedRunningTime="2026-09-11 14:50:58.33282619 +0000 UTC m=+649.576811848" Sep 11 14:51:05.993643 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:05.993602 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb" Sep 11 14:51:05.994091 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:05.993689 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb" Sep 11 14:51:06.006022 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:06.006000 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb" Sep 11 14:51:06.361997 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:06.361920 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb" Sep 11 14:51:29.421839 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:29.421811 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt"] Sep 11 14:51:29.425179 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:29.425158 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt" Sep 11 14:51:29.428285 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:29.428267 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 11 14:51:29.441747 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:29.441720 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt"] Sep 11 14:51:29.490812 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:29.490785 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4acab8e-4026-4d69-9820-1165df545af3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt\" (UID: \"e4acab8e-4026-4d69-9820-1165df545af3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt" Sep 11 14:51:29.490915 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:29.490819 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4acab8e-4026-4d69-9820-1165df545af3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt\" (UID: \"e4acab8e-4026-4d69-9820-1165df545af3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt" Sep 11 14:51:29.490915 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:29.490847 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4acab8e-4026-4d69-9820-1165df545af3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt\" (UID: \"e4acab8e-4026-4d69-9820-1165df545af3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt" Sep 11 14:51:29.491005 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:29.490932 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnnss\" (UniqueName: \"kubernetes.io/projected/e4acab8e-4026-4d69-9820-1165df545af3-kube-api-access-wnnss\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt\" (UID: \"e4acab8e-4026-4d69-9820-1165df545af3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt" Sep 11 14:51:29.491005 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:29.490997 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4acab8e-4026-4d69-9820-1165df545af3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt\" (UID: \"e4acab8e-4026-4d69-9820-1165df545af3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt" Sep 11 14:51:29.491088 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:29.491027 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4acab8e-4026-4d69-9820-1165df545af3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt\" (UID: \"e4acab8e-4026-4d69-9820-1165df545af3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt" Sep 11 14:51:29.592039 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:29.592008 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4acab8e-4026-4d69-9820-1165df545af3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt\" (UID: \"e4acab8e-4026-4d69-9820-1165df545af3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt" Sep 11 14:51:29.592159 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:29.592045 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4acab8e-4026-4d69-9820-1165df545af3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt\" (UID: \"e4acab8e-4026-4d69-9820-1165df545af3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt" Sep 11 14:51:29.592159 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:29.592101 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4acab8e-4026-4d69-9820-1165df545af3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt\" (UID: \"e4acab8e-4026-4d69-9820-1165df545af3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt" Sep 11 14:51:29.592159 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:29.592135 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4acab8e-4026-4d69-9820-1165df545af3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt\" (UID: \"e4acab8e-4026-4d69-9820-1165df545af3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt" Sep 11 14:51:29.592323 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:29.592168 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4acab8e-4026-4d69-9820-1165df545af3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt\" (UID: \"e4acab8e-4026-4d69-9820-1165df545af3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt" Sep 11 14:51:29.592323 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:29.592214 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnnss\" (UniqueName: \"kubernetes.io/projected/e4acab8e-4026-4d69-9820-1165df545af3-kube-api-access-wnnss\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt\" (UID: \"e4acab8e-4026-4d69-9820-1165df545af3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt" Sep 11 14:51:29.592473 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:29.592447 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4acab8e-4026-4d69-9820-1165df545af3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt\" (UID: \"e4acab8e-4026-4d69-9820-1165df545af3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt" Sep 11 14:51:29.592585 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:29.592492 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4acab8e-4026-4d69-9820-1165df545af3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt\" (UID: \"e4acab8e-4026-4d69-9820-1165df545af3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt" Sep 11 14:51:29.592585 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:29.592530 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4acab8e-4026-4d69-9820-1165df545af3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt\" (UID: \"e4acab8e-4026-4d69-9820-1165df545af3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt" Sep 11 14:51:29.594310 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:29.594286 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4acab8e-4026-4d69-9820-1165df545af3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt\" (UID: \"e4acab8e-4026-4d69-9820-1165df545af3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt" Sep 11 14:51:29.594566 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:29.594543 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4acab8e-4026-4d69-9820-1165df545af3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt\" (UID: \"e4acab8e-4026-4d69-9820-1165df545af3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt" Sep 11 14:51:29.603751 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:29.603730 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnnss\" (UniqueName: \"kubernetes.io/projected/e4acab8e-4026-4d69-9820-1165df545af3-kube-api-access-wnnss\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt\" (UID: \"e4acab8e-4026-4d69-9820-1165df545af3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt" Sep 11 14:51:29.735599 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:29.735524 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt" Sep 11 14:51:30.017042 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:30.016966 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt"] Sep 11 14:51:30.020516 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:51:30.020490 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4acab8e_4026_4d69_9820_1165df545af3.slice/crio-4a290254716886c2c9449694b4f9be1efd219bb180993e1300ad4878d47bc9b1 WatchSource:0}: Error finding container 4a290254716886c2c9449694b4f9be1efd219bb180993e1300ad4878d47bc9b1: Status 404 returned error can't find the container with id 4a290254716886c2c9449694b4f9be1efd219bb180993e1300ad4878d47bc9b1 Sep 11 14:51:30.470431 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:30.470393 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt" event={"ID":"e4acab8e-4026-4d69-9820-1165df545af3","Type":"ContainerStarted","Data":"bdca27c7a309601c37357240305098aede32886fa02354f5a06f8395598c6fa8"} Sep 11 14:51:30.470880 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:30.470475 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt" event={"ID":"e4acab8e-4026-4d69-9820-1165df545af3","Type":"ContainerStarted","Data":"4a290254716886c2c9449694b4f9be1efd219bb180993e1300ad4878d47bc9b1"} Sep 11 14:51:30.490897 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:30.490852 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt" podStartSLOduration=1.306677523 podStartE2EDuration="1.490838447s" totalImagesPullingTime="184.160924ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:51:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:51:30.488492648 +0000 UTC m=+681.732478330" watchObservedRunningTime="2026-09-11 14:51:30.490838447 +0000 UTC m=+681.734824106" Sep 11 14:51:39.735771 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:39.735733 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt" Sep 11 14:51:39.736181 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:39.735792 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt" Sep 11 14:51:39.747920 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:39.747892 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt" Sep 11 14:51:40.531343 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:40.531315 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt" Sep 11 14:51:44.978603 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:44.978571 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8"] Sep 11 14:51:44.981956 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:44.981936 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8" Sep 11 14:51:44.985217 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:44.985197 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 11 14:51:44.993947 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:44.993926 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8"] Sep 11 14:51:45.117955 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:45.117929 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23eb9fe9-3dac-4380-a381-d0f08e595303-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjrd8\" (UID: \"23eb9fe9-3dac-4380-a381-d0f08e595303\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8" Sep 11 14:51:45.118079 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:45.117960 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23eb9fe9-3dac-4380-a381-d0f08e595303-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjrd8\" (UID: \"23eb9fe9-3dac-4380-a381-d0f08e595303\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8" Sep 11 14:51:45.118079 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:45.117989 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plsl9\" (UniqueName: \"kubernetes.io/projected/23eb9fe9-3dac-4380-a381-d0f08e595303-kube-api-access-plsl9\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjrd8\" (UID: \"23eb9fe9-3dac-4380-a381-d0f08e595303\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8" Sep 11 14:51:45.118204 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:45.118082 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23eb9fe9-3dac-4380-a381-d0f08e595303-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjrd8\" (UID: \"23eb9fe9-3dac-4380-a381-d0f08e595303\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8" Sep 11 14:51:45.118204 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:45.118132 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23eb9fe9-3dac-4380-a381-d0f08e595303-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjrd8\" (UID: \"23eb9fe9-3dac-4380-a381-d0f08e595303\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8" Sep 11 14:51:45.118204 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:45.118173 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23eb9fe9-3dac-4380-a381-d0f08e595303-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjrd8\" (UID: \"23eb9fe9-3dac-4380-a381-d0f08e595303\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8" Sep 11 14:51:45.218793 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:45.218760 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23eb9fe9-3dac-4380-a381-d0f08e595303-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjrd8\" (UID: \"23eb9fe9-3dac-4380-a381-d0f08e595303\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8" Sep 11 14:51:45.218950 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:45.218800 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23eb9fe9-3dac-4380-a381-d0f08e595303-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjrd8\" (UID: \"23eb9fe9-3dac-4380-a381-d0f08e595303\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8" Sep 11 14:51:45.218950 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:45.218829 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plsl9\" (UniqueName: \"kubernetes.io/projected/23eb9fe9-3dac-4380-a381-d0f08e595303-kube-api-access-plsl9\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjrd8\" (UID: \"23eb9fe9-3dac-4380-a381-d0f08e595303\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8" Sep 11 14:51:45.218950 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:45.218864 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23eb9fe9-3dac-4380-a381-d0f08e595303-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjrd8\" (UID: \"23eb9fe9-3dac-4380-a381-d0f08e595303\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8" Sep 11 14:51:45.218950 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:45.218901 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23eb9fe9-3dac-4380-a381-d0f08e595303-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjrd8\" (UID: \"23eb9fe9-3dac-4380-a381-d0f08e595303\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8" Sep 11 14:51:45.219158 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:45.219032 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23eb9fe9-3dac-4380-a381-d0f08e595303-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjrd8\" (UID: \"23eb9fe9-3dac-4380-a381-d0f08e595303\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8" Sep 11 14:51:45.219294 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:45.219276 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23eb9fe9-3dac-4380-a381-d0f08e595303-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjrd8\" (UID: \"23eb9fe9-3dac-4380-a381-d0f08e595303\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8" Sep 11 14:51:45.219384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:45.219353 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23eb9fe9-3dac-4380-a381-d0f08e595303-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjrd8\" (UID: \"23eb9fe9-3dac-4380-a381-d0f08e595303\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8" Sep 11 14:51:45.219502 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:45.219397 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23eb9fe9-3dac-4380-a381-d0f08e595303-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjrd8\" (UID: \"23eb9fe9-3dac-4380-a381-d0f08e595303\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8" Sep 11 14:51:45.221081 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:45.221058 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23eb9fe9-3dac-4380-a381-d0f08e595303-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjrd8\" (UID: \"23eb9fe9-3dac-4380-a381-d0f08e595303\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8" Sep 11 14:51:45.221330 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:45.221315 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23eb9fe9-3dac-4380-a381-d0f08e595303-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjrd8\" (UID: \"23eb9fe9-3dac-4380-a381-d0f08e595303\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8" Sep 11 14:51:45.228487 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:45.228463 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plsl9\" (UniqueName: \"kubernetes.io/projected/23eb9fe9-3dac-4380-a381-d0f08e595303-kube-api-access-plsl9\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jjrd8\" (UID: \"23eb9fe9-3dac-4380-a381-d0f08e595303\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8" Sep 11 14:51:45.294076 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:45.294024 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8" Sep 11 14:51:45.568559 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:45.568532 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8"] Sep 11 14:51:45.570510 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:51:45.570483 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23eb9fe9_3dac_4380_a381_d0f08e595303.slice/crio-de64f7b207551f675bcf597c1009fa6e8d64df1e84f98349af3078b2d1b66199 WatchSource:0}: Error finding container de64f7b207551f675bcf597c1009fa6e8d64df1e84f98349af3078b2d1b66199: Status 404 returned error can't find the container with id de64f7b207551f675bcf597c1009fa6e8d64df1e84f98349af3078b2d1b66199 Sep 11 14:51:46.551733 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:46.551687 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8" event={"ID":"23eb9fe9-3dac-4380-a381-d0f08e595303","Type":"ContainerStarted","Data":"8841718b47d1c0a9d20f7d63b4966d623576474419525980383e9995cdf27c04"} Sep 11 14:51:46.552102 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:46.551751 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8" event={"ID":"23eb9fe9-3dac-4380-a381-d0f08e595303","Type":"ContainerStarted","Data":"de64f7b207551f675bcf597c1009fa6e8d64df1e84f98349af3078b2d1b66199"} Sep 11 14:51:46.573635 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:46.573592 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8" podStartSLOduration=2.390292397 podStartE2EDuration="2.573576436s" totalImagesPullingTime="183.284039ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:51:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:51:46.572083655 +0000 UTC m=+697.816069314" watchObservedRunningTime="2026-09-11 14:51:46.573576436 +0000 UTC m=+697.817562094" Sep 11 14:51:55.294584 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:55.294541 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8" Sep 11 14:51:55.294999 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:55.294604 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8" Sep 11 14:51:55.307033 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:55.307013 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8" Sep 11 14:51:55.610201 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:51:55.610120 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jjrd8" Sep 11 14:52:53.774149 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:52:53.774107 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65c5c88497-kbb4p"] Sep 11 14:52:53.774679 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:52:53.774332 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-65c5c88497-kbb4p" podUID="f125bd03-314d-4987-a471-bdbe74de3feb" containerName="authorino" containerID="cri-o://0b2d8a9dc5e6d5d4bbab924d186672ad292a897eff657f2d5a0afa55dab22790" gracePeriod=30 Sep 11 14:52:53.902005 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:52:53.901966 2583 generic.go:356] "Generic (PLEG): container finished" podID="f125bd03-314d-4987-a471-bdbe74de3feb" containerID="0b2d8a9dc5e6d5d4bbab924d186672ad292a897eff657f2d5a0afa55dab22790" exitCode=0 Sep 11 14:52:53.902194 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:52:53.902006 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65c5c88497-kbb4p" event={"ID":"f125bd03-314d-4987-a471-bdbe74de3feb","Type":"ContainerDied","Data":"0b2d8a9dc5e6d5d4bbab924d186672ad292a897eff657f2d5a0afa55dab22790"} Sep 11 14:52:54.011798 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:52:54.011772 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65c5c88497-kbb4p" Sep 11 14:52:54.109374 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:52:54.109281 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ns2tq\" (UniqueName: \"kubernetes.io/projected/f125bd03-314d-4987-a471-bdbe74de3feb-kube-api-access-ns2tq\") pod \"f125bd03-314d-4987-a471-bdbe74de3feb\" (UID: \"f125bd03-314d-4987-a471-bdbe74de3feb\") " Sep 11 14:52:54.109374 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:52:54.109317 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f125bd03-314d-4987-a471-bdbe74de3feb-tls-cert\") pod \"f125bd03-314d-4987-a471-bdbe74de3feb\" (UID: \"f125bd03-314d-4987-a471-bdbe74de3feb\") " Sep 11 14:52:54.111630 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:52:54.111601 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f125bd03-314d-4987-a471-bdbe74de3feb-kube-api-access-ns2tq" (OuterVolumeSpecName: "kube-api-access-ns2tq") pod "f125bd03-314d-4987-a471-bdbe74de3feb" (UID: "f125bd03-314d-4987-a471-bdbe74de3feb"). InnerVolumeSpecName "kube-api-access-ns2tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:52:54.120270 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:52:54.120240 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f125bd03-314d-4987-a471-bdbe74de3feb-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "f125bd03-314d-4987-a471-bdbe74de3feb" (UID: "f125bd03-314d-4987-a471-bdbe74de3feb"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:52:54.210554 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:52:54.210499 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns2tq\" (UniqueName: \"kubernetes.io/projected/f125bd03-314d-4987-a471-bdbe74de3feb-kube-api-access-ns2tq\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:52:54.210554 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:52:54.210553 2583 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f125bd03-314d-4987-a471-bdbe74de3feb-tls-cert\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:52:54.908426 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:52:54.908396 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65c5c88497-kbb4p" Sep 11 14:52:54.908426 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:52:54.908421 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65c5c88497-kbb4p" event={"ID":"f125bd03-314d-4987-a471-bdbe74de3feb","Type":"ContainerDied","Data":"dcb75c56403414c8be425ac47688d7db9d038fe987993de3b20853b2567d9ece"} Sep 11 14:52:54.908907 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:52:54.908466 2583 scope.go:117] "RemoveContainer" containerID="0b2d8a9dc5e6d5d4bbab924d186672ad292a897eff657f2d5a0afa55dab22790" Sep 11 14:52:54.947573 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:52:54.947524 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65c5c88497-kbb4p"] Sep 11 14:52:54.960283 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:52:54.960251 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-65c5c88497-kbb4p"] Sep 11 14:52:55.330084 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:52:55.330047 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f125bd03-314d-4987-a471-bdbe74de3feb" path="/var/lib/kubelet/pods/f125bd03-314d-4987-a471-bdbe74de3feb/volumes" Sep 11 14:53:14.880739 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:14.880692 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7"] Sep 11 14:53:14.881101 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:14.881041 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f125bd03-314d-4987-a471-bdbe74de3feb" containerName="authorino" Sep 11 14:53:14.881101 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:14.881053 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f125bd03-314d-4987-a471-bdbe74de3feb" containerName="authorino" Sep 11 14:53:14.881173 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:14.881119 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f125bd03-314d-4987-a471-bdbe74de3feb" containerName="authorino" Sep 11 14:53:14.885141 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:14.885123 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:14.887933 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:14.887908 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-837632-openshift-default-dockercfg-dx7mp\"" Sep 11 14:53:14.907165 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:14.907117 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7"] Sep 11 14:53:14.983110 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:14.983071 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istiod-ca-cert\") pod \"e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:14.983110 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:14.983124 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istio-envoy\") pod \"e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:14.983340 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:14.983146 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istio-podinfo\") pod \"e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:14.983340 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:14.983170 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istio-token\") pod \"e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:14.983340 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:14.983244 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-workload-socket\") pod \"e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:14.983340 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:14.983280 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txt6c\" (UniqueName: \"kubernetes.io/projected/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-kube-api-access-txt6c\") pod \"e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:14.983340 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:14.983298 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-workload-certs\") pod \"e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:14.983511 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:14.983385 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-credential-socket\") pod \"e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:14.983511 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:14.983424 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istio-data\") pod \"e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:15.064867 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.064827 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9"] Sep 11 14:53:15.068322 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.068302 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.071606 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.071576 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-e7ca56-openshift-default-dockercfg-kpms7\"" Sep 11 14:53:15.084739 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.084690 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-workload-certs\") pod \"e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:15.084921 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.084770 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-credential-socket\") pod \"e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:15.084921 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.084801 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istio-data\") pod \"e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:15.084921 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.084831 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istiod-ca-cert\") pod \"e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:15.084921 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.084871 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9"] Sep 11 14:53:15.085139 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.084875 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istio-envoy\") pod \"e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:15.085139 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.084971 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istio-podinfo\") pod \"e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:15.085139 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.084999 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istio-token\") pod \"e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:15.085139 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.085059 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-workload-socket\") pod \"e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:15.085139 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.085113 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-workload-certs\") pod \"e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:15.085139 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.085124 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txt6c\" (UniqueName: \"kubernetes.io/projected/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-kube-api-access-txt6c\") pod \"e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:15.085607 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.085580 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istiod-ca-cert\") pod \"e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:15.085771 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.085749 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-credential-socket\") pod \"e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:15.085889 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.085840 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-workload-socket\") pod \"e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:15.085951 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.085890 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istio-data\") pod \"e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:15.087740 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.087700 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istio-envoy\") pod \"e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:15.088028 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.088010 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istio-podinfo\") pod \"e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:15.093542 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.093514 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istio-token\") pod \"e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:15.093674 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.093522 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txt6c\" (UniqueName: \"kubernetes.io/projected/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-kube-api-access-txt6c\") pod \"e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:15.186810 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.186727 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istio-token\") pod \"e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.186810 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.186799 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istiod-ca-cert\") pod \"e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.187009 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.186847 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-workload-certs\") pod \"e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.187009 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.186878 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istio-envoy\") pod \"e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.187009 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.186900 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc4cx\" (UniqueName: \"kubernetes.io/projected/91ab0283-f94d-45bb-9b61-ad4d927d8b68-kube-api-access-wc4cx\") pod \"e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.187009 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.186942 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istio-podinfo\") pod \"e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.187009 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.186981 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-workload-socket\") pod \"e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.187189 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.187030 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istio-data\") pod \"e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.187189 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.187056 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-credential-socket\") pod \"e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.195544 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.195508 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:15.288241 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.288201 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-credential-socket\") pod \"e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.288421 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.288338 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istio-token\") pod \"e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.288421 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.288358 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istiod-ca-cert\") pod \"e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.288421 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.288392 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-workload-certs\") pod \"e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.288578 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.288518 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istio-envoy\") pod \"e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.288624 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.288574 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc4cx\" (UniqueName: \"kubernetes.io/projected/91ab0283-f94d-45bb-9b61-ad4d927d8b68-kube-api-access-wc4cx\") pod \"e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.288674 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.288622 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istio-podinfo\") pod \"e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.288674 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.288645 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-credential-socket\") pod \"e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.288674 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.288656 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-workload-socket\") pod \"e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.288884 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.288749 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istio-data\") pod \"e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.288884 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.288837 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-workload-certs\") pod \"e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.289042 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.289017 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-workload-socket\") pod \"e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.289182 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.289079 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istio-data\") pod \"e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.289182 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.289079 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istiod-ca-cert\") pod \"e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.290984 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.290959 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istio-envoy\") pod \"e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.291167 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.291144 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istio-podinfo\") pod \"e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.297210 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.297178 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc4cx\" (UniqueName: \"kubernetes.io/projected/91ab0283-f94d-45bb-9b61-ad4d927d8b68-kube-api-access-wc4cx\") pod \"e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.297604 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.297585 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istio-token\") pod \"e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.379784 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.379750 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:15.482583 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.482493 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7"] Sep 11 14:53:15.485464 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:53:15.485439 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f745a9f_d6ba_4629_b8c0_8faacfbd4845.slice/crio-c83fa480ece593bb4a413af4acfd61708f507a5c212331817ade3161034de61e WatchSource:0}: Error finding container c83fa480ece593bb4a413af4acfd61708f507a5c212331817ade3161034de61e: Status 404 returned error can't find the container with id c83fa480ece593bb4a413af4acfd61708f507a5c212331817ade3161034de61e Sep 11 14:53:15.487544 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.487508 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:53:15.487643 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.487595 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:53:15.487751 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.487638 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:53:15.661598 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.661564 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9"] Sep 11 14:53:15.662231 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:53:15.662202 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ab0283_f94d_45bb_9b61_ad4d927d8b68.slice/crio-4e75b77f3dabcf5ff279d85ddab069aa8a1b4657c8fba1afe0668789df19244a WatchSource:0}: Error finding container 4e75b77f3dabcf5ff279d85ddab069aa8a1b4657c8fba1afe0668789df19244a: Status 404 returned error can't find the container with id 4e75b77f3dabcf5ff279d85ddab069aa8a1b4657c8fba1afe0668789df19244a Sep 11 14:53:15.664457 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.664422 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:53:15.664541 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.664490 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:53:15.664541 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:15.664519 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:53:16.025386 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:16.025346 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" event={"ID":"91ab0283-f94d-45bb-9b61-ad4d927d8b68","Type":"ContainerStarted","Data":"315a9c33c41e8e74ae348e51248b8be0e053a6fd86fc945fd0f45cd81ebe646f"} Sep 11 14:53:16.025883 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:16.025417 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" event={"ID":"91ab0283-f94d-45bb-9b61-ad4d927d8b68","Type":"ContainerStarted","Data":"4e75b77f3dabcf5ff279d85ddab069aa8a1b4657c8fba1afe0668789df19244a"} Sep 11 14:53:16.027091 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:16.027065 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" event={"ID":"9f745a9f-d6ba-4629-b8c0-8faacfbd4845","Type":"ContainerStarted","Data":"27a4bd6866cdf80cc2ce89ec8dda169627b693d5a2e4e5292cbe4b9c49628f6d"} Sep 11 14:53:16.027168 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:16.027109 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" event={"ID":"9f745a9f-d6ba-4629-b8c0-8faacfbd4845","Type":"ContainerStarted","Data":"c83fa480ece593bb4a413af4acfd61708f507a5c212331817ade3161034de61e"} Sep 11 14:53:16.056282 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:16.056227 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" podStartSLOduration=1.056204586 podStartE2EDuration="1.056204586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:16.055368619 +0000 UTC m=+787.299354278" watchObservedRunningTime="2026-09-11 14:53:16.056204586 +0000 UTC m=+787.300190245" Sep 11 14:53:16.082924 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:16.082863 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" podStartSLOduration=2.082845256 podStartE2EDuration="2.082845256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:16.080475563 +0000 UTC m=+787.324461223" watchObservedRunningTime="2026-09-11 14:53:16.082845256 +0000 UTC m=+787.326830915" Sep 11 14:53:16.196226 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:16.196179 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:16.197611 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:16.197581 2583 patch_prober.go:28] interesting pod/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Sep 11 14:53:16.197766 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:16.197647 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" podUID="9f745a9f-d6ba-4629-b8c0-8faacfbd4845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Sep 11 14:53:16.380654 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:16.380619 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:16.386657 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:16.386629 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:17.033168 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:17.033132 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:17.034090 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:17.034072 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:17.199974 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:17.199945 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:18.037690 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:18.037655 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:18.038728 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:18.038682 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:21.850933 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:21.850890 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f"] Sep 11 14:53:21.855607 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:21.855575 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f" Sep 11 14:53:21.859414 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:21.859385 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-e7ca56-dockercfg-54ns2\"" Sep 11 14:53:21.859414 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:21.859385 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-e7ca56\"" Sep 11 14:53:21.866633 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:21.866598 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f"] Sep 11 14:53:21.958831 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:21.958790 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743de4e7-8ce3-4871-9d50-6f05b34bb27b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f\" (UID: \"743de4e7-8ce3-4871-9d50-6f05b34bb27b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f" Sep 11 14:53:21.959014 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:21.958958 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d5ns\" (UniqueName: \"kubernetes.io/projected/743de4e7-8ce3-4871-9d50-6f05b34bb27b-kube-api-access-5d5ns\") pod \"payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f\" (UID: \"743de4e7-8ce3-4871-9d50-6f05b34bb27b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f" Sep 11 14:53:22.060298 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:22.060265 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d5ns\" (UniqueName: \"kubernetes.io/projected/743de4e7-8ce3-4871-9d50-6f05b34bb27b-kube-api-access-5d5ns\") pod \"payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f\" (UID: \"743de4e7-8ce3-4871-9d50-6f05b34bb27b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f" Sep 11 14:53:22.060463 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:22.060319 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743de4e7-8ce3-4871-9d50-6f05b34bb27b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f\" (UID: \"743de4e7-8ce3-4871-9d50-6f05b34bb27b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f" Sep 11 14:53:22.060996 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:22.060977 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743de4e7-8ce3-4871-9d50-6f05b34bb27b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f\" (UID: \"743de4e7-8ce3-4871-9d50-6f05b34bb27b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f" Sep 11 14:53:22.075083 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:22.075051 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d5ns\" (UniqueName: \"kubernetes.io/projected/743de4e7-8ce3-4871-9d50-6f05b34bb27b-kube-api-access-5d5ns\") pod \"payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f\" (UID: \"743de4e7-8ce3-4871-9d50-6f05b34bb27b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f" Sep 11 14:53:22.168700 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:22.168659 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f" Sep 11 14:53:22.450170 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:22.450143 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f"] Sep 11 14:53:22.452521 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:53:22.452491 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod743de4e7_8ce3_4871_9d50_6f05b34bb27b.slice/crio-c3d7f2d8af0808d4d072ded9907f3375d927339857130d48ba2cf88d115a46f2 WatchSource:0}: Error finding container c3d7f2d8af0808d4d072ded9907f3375d927339857130d48ba2cf88d115a46f2: Status 404 returned error can't find the container with id c3d7f2d8af0808d4d072ded9907f3375d927339857130d48ba2cf88d115a46f2 Sep 11 14:53:23.066384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.066347 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f" event={"ID":"743de4e7-8ce3-4871-9d50-6f05b34bb27b","Type":"ContainerStarted","Data":"15c841ce5b7ae0889627e63f3c05ecfcbb5619cbd3c352dccbe52f6e2eca4f0c"} Sep 11 14:53:23.066892 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.066405 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f" event={"ID":"743de4e7-8ce3-4871-9d50-6f05b34bb27b","Type":"ContainerStarted","Data":"c3d7f2d8af0808d4d072ded9907f3375d927339857130d48ba2cf88d115a46f2"} Sep 11 14:53:23.066892 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.066474 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f" Sep 11 14:53:23.088209 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.088147 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f" podStartSLOduration=2.088127657 podStartE2EDuration="2.088127657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:23.087386837 +0000 UTC m=+794.331372496" watchObservedRunningTime="2026-09-11 14:53:23.088127657 +0000 UTC m=+794.332113318" Sep 11 14:53:23.552996 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.552960 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf"] Sep 11 14:53:23.556680 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.556657 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.559618 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.559597 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-3219e3-openshift-default-dockercfg-mp967\"" Sep 11 14:53:23.572921 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.572899 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf"] Sep 11 14:53:23.675472 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.675437 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26130197-f0e3-4871-abb4-58a1f6331446-istio-podinfo\") pod \"e2e-autores-3219e3-openshift-default-587759d66d-9zqzf\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.675757 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.675730 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr9ck\" (UniqueName: \"kubernetes.io/projected/26130197-f0e3-4871-abb4-58a1f6331446-kube-api-access-xr9ck\") pod \"e2e-autores-3219e3-openshift-default-587759d66d-9zqzf\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.675870 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.675822 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26130197-f0e3-4871-abb4-58a1f6331446-istiod-ca-cert\") pod \"e2e-autores-3219e3-openshift-default-587759d66d-9zqzf\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.675932 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.675877 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-istio-data\") pod \"e2e-autores-3219e3-openshift-default-587759d66d-9zqzf\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.675986 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.675952 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-credential-socket\") pod \"e2e-autores-3219e3-openshift-default-587759d66d-9zqzf\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.676045 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.676006 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-workload-socket\") pod \"e2e-autores-3219e3-openshift-default-587759d66d-9zqzf\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.676100 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.676040 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-workload-certs\") pod \"e2e-autores-3219e3-openshift-default-587759d66d-9zqzf\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.676100 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.676069 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26130197-f0e3-4871-abb4-58a1f6331446-istio-token\") pod \"e2e-autores-3219e3-openshift-default-587759d66d-9zqzf\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.676197 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.676129 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-istio-envoy\") pod \"e2e-autores-3219e3-openshift-default-587759d66d-9zqzf\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.777418 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.777384 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr9ck\" (UniqueName: \"kubernetes.io/projected/26130197-f0e3-4871-abb4-58a1f6331446-kube-api-access-xr9ck\") pod \"e2e-autores-3219e3-openshift-default-587759d66d-9zqzf\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.777614 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.777422 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26130197-f0e3-4871-abb4-58a1f6331446-istiod-ca-cert\") pod \"e2e-autores-3219e3-openshift-default-587759d66d-9zqzf\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.777614 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.777465 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-istio-data\") pod \"e2e-autores-3219e3-openshift-default-587759d66d-9zqzf\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.777614 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.777526 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-credential-socket\") pod \"e2e-autores-3219e3-openshift-default-587759d66d-9zqzf\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.777614 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.777567 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-workload-socket\") pod \"e2e-autores-3219e3-openshift-default-587759d66d-9zqzf\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.777614 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.777594 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-workload-certs\") pod \"e2e-autores-3219e3-openshift-default-587759d66d-9zqzf\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.777923 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.777622 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26130197-f0e3-4871-abb4-58a1f6331446-istio-token\") pod \"e2e-autores-3219e3-openshift-default-587759d66d-9zqzf\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.777923 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.777668 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-istio-envoy\") pod \"e2e-autores-3219e3-openshift-default-587759d66d-9zqzf\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.777923 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.777745 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26130197-f0e3-4871-abb4-58a1f6331446-istio-podinfo\") pod \"e2e-autores-3219e3-openshift-default-587759d66d-9zqzf\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.778083 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.778005 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-credential-socket\") pod \"e2e-autores-3219e3-openshift-default-587759d66d-9zqzf\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.778207 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.778079 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-workload-socket\") pod \"e2e-autores-3219e3-openshift-default-587759d66d-9zqzf\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.778207 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.778177 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-workload-certs\") pod \"e2e-autores-3219e3-openshift-default-587759d66d-9zqzf\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.778318 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.778257 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-istio-data\") pod \"e2e-autores-3219e3-openshift-default-587759d66d-9zqzf\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.778318 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.778286 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26130197-f0e3-4871-abb4-58a1f6331446-istiod-ca-cert\") pod \"e2e-autores-3219e3-openshift-default-587759d66d-9zqzf\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.780082 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.780056 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-istio-envoy\") pod \"e2e-autores-3219e3-openshift-default-587759d66d-9zqzf\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.780259 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.780239 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26130197-f0e3-4871-abb4-58a1f6331446-istio-podinfo\") pod \"e2e-autores-3219e3-openshift-default-587759d66d-9zqzf\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.786670 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.786651 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26130197-f0e3-4871-abb4-58a1f6331446-istio-token\") pod \"e2e-autores-3219e3-openshift-default-587759d66d-9zqzf\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.787088 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.787068 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr9ck\" (UniqueName: \"kubernetes.io/projected/26130197-f0e3-4871-abb4-58a1f6331446-kube-api-access-xr9ck\") pod \"e2e-autores-3219e3-openshift-default-587759d66d-9zqzf\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:23.867674 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:23.867607 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:24.150355 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:24.150324 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf"] Sep 11 14:53:24.152230 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:53:24.152202 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26130197_f0e3_4871_abb4_58a1f6331446.slice/crio-03115373058e4e9a70ea4d2f36284cb5e0ca1514b67298a03be5ddf7c81d9a7c WatchSource:0}: Error finding container 03115373058e4e9a70ea4d2f36284cb5e0ca1514b67298a03be5ddf7c81d9a7c: Status 404 returned error can't find the container with id 03115373058e4e9a70ea4d2f36284cb5e0ca1514b67298a03be5ddf7c81d9a7c Sep 11 14:53:24.154438 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:24.154408 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:53:24.154513 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:24.154467 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:53:24.154513 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:24.154496 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:53:25.079281 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:25.079250 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" event={"ID":"26130197-f0e3-4871-abb4-58a1f6331446","Type":"ContainerStarted","Data":"73fb9bb192d89744f9d7008d4565d56a8274e351d767bf4d8cd2f4d1a7e67216"} Sep 11 14:53:25.079448 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:25.079300 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" event={"ID":"26130197-f0e3-4871-abb4-58a1f6331446","Type":"ContainerStarted","Data":"03115373058e4e9a70ea4d2f36284cb5e0ca1514b67298a03be5ddf7c81d9a7c"} Sep 11 14:53:25.101871 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:25.101827 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" podStartSLOduration=2.1018114 podStartE2EDuration="2.1018114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:25.100425712 +0000 UTC m=+796.344411371" watchObservedRunningTime="2026-09-11 14:53:25.1018114 +0000 UTC m=+796.345797058" Sep 11 14:53:25.867962 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:25.867924 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:25.872884 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:25.872861 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:26.084383 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:26.084353 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:26.085282 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:26.085265 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:26.384089 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:26.384056 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj"] Sep 11 14:53:26.387758 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:26.387741 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj" Sep 11 14:53:26.393461 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:26.393438 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-362f7a\"" Sep 11 14:53:26.404681 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:26.404654 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj"] Sep 11 14:53:26.505515 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:26.505478 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e861242-c72a-48df-a49f-654e107cbd3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj\" (UID: \"8e861242-c72a-48df-a49f-654e107cbd3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj" Sep 11 14:53:26.505693 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:26.505553 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx9sx\" (UniqueName: \"kubernetes.io/projected/8e861242-c72a-48df-a49f-654e107cbd3c-kube-api-access-hx9sx\") pod \"payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj\" (UID: \"8e861242-c72a-48df-a49f-654e107cbd3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj" Sep 11 14:53:26.606482 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:26.606444 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e861242-c72a-48df-a49f-654e107cbd3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj\" (UID: \"8e861242-c72a-48df-a49f-654e107cbd3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj" Sep 11 14:53:26.606673 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:26.606515 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx9sx\" (UniqueName: \"kubernetes.io/projected/8e861242-c72a-48df-a49f-654e107cbd3c-kube-api-access-hx9sx\") pod \"payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj\" (UID: \"8e861242-c72a-48df-a49f-654e107cbd3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj" Sep 11 14:53:26.607515 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:26.607488 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e861242-c72a-48df-a49f-654e107cbd3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj\" (UID: \"8e861242-c72a-48df-a49f-654e107cbd3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj" Sep 11 14:53:26.632123 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:26.632091 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx9sx\" (UniqueName: \"kubernetes.io/projected/8e861242-c72a-48df-a49f-654e107cbd3c-kube-api-access-hx9sx\") pod \"payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj\" (UID: \"8e861242-c72a-48df-a49f-654e107cbd3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj" Sep 11 14:53:26.690694 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:26.690624 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc"] Sep 11 14:53:26.694190 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:26.694174 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc" Sep 11 14:53:26.697923 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:26.697898 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj" Sep 11 14:53:26.698858 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:26.698838 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-362f7a-dockercfg-2vrjg\"" Sep 11 14:53:26.709901 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:26.709873 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc"] Sep 11 14:53:26.812417 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:26.809202 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gzk4\" (UniqueName: \"kubernetes.io/projected/94105113-4ed4-4985-86bf-911aa03ba7fd-kube-api-access-8gzk4\") pod \"payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc\" (UID: \"94105113-4ed4-4985-86bf-911aa03ba7fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc" Sep 11 14:53:26.812417 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:26.809280 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94105113-4ed4-4985-86bf-911aa03ba7fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc\" (UID: \"94105113-4ed4-4985-86bf-911aa03ba7fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc" Sep 11 14:53:26.909814 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:26.909784 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gzk4\" (UniqueName: \"kubernetes.io/projected/94105113-4ed4-4985-86bf-911aa03ba7fd-kube-api-access-8gzk4\") pod \"payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc\" (UID: \"94105113-4ed4-4985-86bf-911aa03ba7fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc" Sep 11 14:53:26.910211 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:26.909822 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94105113-4ed4-4985-86bf-911aa03ba7fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc\" (UID: \"94105113-4ed4-4985-86bf-911aa03ba7fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc" Sep 11 14:53:26.910536 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:26.910509 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94105113-4ed4-4985-86bf-911aa03ba7fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc\" (UID: \"94105113-4ed4-4985-86bf-911aa03ba7fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc" Sep 11 14:53:26.919238 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:26.919209 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gzk4\" (UniqueName: \"kubernetes.io/projected/94105113-4ed4-4985-86bf-911aa03ba7fd-kube-api-access-8gzk4\") pod \"payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc\" (UID: \"94105113-4ed4-4985-86bf-911aa03ba7fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc" Sep 11 14:53:26.992083 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:26.991971 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj"] Sep 11 14:53:26.996422 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:53:26.996395 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e861242_c72a_48df_a49f_654e107cbd3c.slice/crio-4e57262ab8320f7aa174a142c056489af126af8d2e52b075d1c22057200a4272 WatchSource:0}: Error finding container 4e57262ab8320f7aa174a142c056489af126af8d2e52b075d1c22057200a4272: Status 404 returned error can't find the container with id 4e57262ab8320f7aa174a142c056489af126af8d2e52b075d1c22057200a4272 Sep 11 14:53:27.005823 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:27.005801 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc" Sep 11 14:53:27.096305 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:27.095949 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj" event={"ID":"8e861242-c72a-48df-a49f-654e107cbd3c","Type":"ContainerStarted","Data":"4e57262ab8320f7aa174a142c056489af126af8d2e52b075d1c22057200a4272"} Sep 11 14:53:27.338588 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:27.338432 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc"] Sep 11 14:53:28.102235 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:28.102188 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj" event={"ID":"8e861242-c72a-48df-a49f-654e107cbd3c","Type":"ContainerStarted","Data":"20889530de713cc4f662eff0f35570e6a5fa2bb668d19c038b38a0bb0cf7e4ae"} Sep 11 14:53:28.102736 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:28.102260 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj" Sep 11 14:53:28.103913 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:28.103885 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc" event={"ID":"94105113-4ed4-4985-86bf-911aa03ba7fd","Type":"ContainerStarted","Data":"cadef8c5da8af9a33a19d830785942bd66f7aab86cd37f333f5821359ab574dd"} Sep 11 14:53:28.104053 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:28.103934 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc" event={"ID":"94105113-4ed4-4985-86bf-911aa03ba7fd","Type":"ContainerStarted","Data":"b55a272996f48f3abc0438c803da081f42495d49f850d5ddc10a8ec54faa0189"} Sep 11 14:53:28.104053 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:28.104050 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc" Sep 11 14:53:28.121957 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:28.121891 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj" podStartSLOduration=2.12187252 podStartE2EDuration="2.12187252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:28.120178777 +0000 UTC m=+799.364164448" watchObservedRunningTime="2026-09-11 14:53:28.12187252 +0000 UTC m=+799.365858180" Sep 11 14:53:31.691491 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:31.691439 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc" podStartSLOduration=5.69142178 podStartE2EDuration="5.69142178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:28.146507767 +0000 UTC m=+799.390493427" watchObservedRunningTime="2026-09-11 14:53:31.69142178 +0000 UTC m=+802.935407478" Sep 11 14:53:31.691885 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:31.691690 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq"] Sep 11 14:53:31.695308 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:31.695288 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq" Sep 11 14:53:31.698057 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:31.698035 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-3219e3-dockercfg-5c55m\"" Sep 11 14:53:31.698211 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:31.698055 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-3219e3\"" Sep 11 14:53:31.704670 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:31.704640 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2151993-9943-413a-a5bb-c136052b15dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq\" (UID: \"c2151993-9943-413a-a5bb-c136052b15dc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq" Sep 11 14:53:31.704829 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:31.704752 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdj7l\" (UniqueName: \"kubernetes.io/projected/c2151993-9943-413a-a5bb-c136052b15dc-kube-api-access-sdj7l\") pod \"payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq\" (UID: \"c2151993-9943-413a-a5bb-c136052b15dc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq" Sep 11 14:53:31.706274 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:31.706247 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq"] Sep 11 14:53:31.805871 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:31.805828 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2151993-9943-413a-a5bb-c136052b15dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq\" (UID: \"c2151993-9943-413a-a5bb-c136052b15dc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq" Sep 11 14:53:31.806069 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:31.805922 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdj7l\" (UniqueName: \"kubernetes.io/projected/c2151993-9943-413a-a5bb-c136052b15dc-kube-api-access-sdj7l\") pod \"payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq\" (UID: \"c2151993-9943-413a-a5bb-c136052b15dc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq" Sep 11 14:53:31.807237 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:31.807205 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2151993-9943-413a-a5bb-c136052b15dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq\" (UID: \"c2151993-9943-413a-a5bb-c136052b15dc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq" Sep 11 14:53:31.816132 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:31.815553 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdj7l\" (UniqueName: \"kubernetes.io/projected/c2151993-9943-413a-a5bb-c136052b15dc-kube-api-access-sdj7l\") pod \"payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq\" (UID: \"c2151993-9943-413a-a5bb-c136052b15dc\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq" Sep 11 14:53:32.008642 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:32.008548 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq" Sep 11 14:53:32.333781 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:32.330859 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq"] Sep 11 14:53:32.352252 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:53:32.352222 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2151993_9943_413a_a5bb_c136052b15dc.slice/crio-0a3b8be433d408366c0ee8082218506370c43d40d0c92dd0e7aae3cdc4d412eb WatchSource:0}: Error finding container 0a3b8be433d408366c0ee8082218506370c43d40d0c92dd0e7aae3cdc4d412eb: Status 404 returned error can't find the container with id 0a3b8be433d408366c0ee8082218506370c43d40d0c92dd0e7aae3cdc4d412eb Sep 11 14:53:33.191115 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:33.191078 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq" event={"ID":"c2151993-9943-413a-a5bb-c136052b15dc","Type":"ContainerStarted","Data":"445c5067b4b9eac525b288df0502e52fab49cea26777dbcb2b08002d28bf6fbb"} Sep 11 14:53:33.191585 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:33.191138 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq" event={"ID":"c2151993-9943-413a-a5bb-c136052b15dc","Type":"ContainerStarted","Data":"0a3b8be433d408366c0ee8082218506370c43d40d0c92dd0e7aae3cdc4d412eb"} Sep 11 14:53:33.191585 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:33.191188 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq" Sep 11 14:53:33.210114 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:33.210068 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq" podStartSLOduration=2.210054042 podStartE2EDuration="2.210054042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:33.209131377 +0000 UTC m=+804.453117038" watchObservedRunningTime="2026-09-11 14:53:33.210054042 +0000 UTC m=+804.454039703" Sep 11 14:53:34.074771 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:34.074734 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f" Sep 11 14:53:34.080879 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:34.080840 2583 patch_prober.go:28] interesting pod/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 14:53:34.081054 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:34.080913 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" podUID="9f745a9f-d6ba-4629-b8c0-8faacfbd4845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 14:53:34.710949 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:34.710911 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f722bf59-79c869485-hdk7v"] Sep 11 14:53:34.714453 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:34.714436 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f722bf59-79c869485-hdk7v" Sep 11 14:53:34.717194 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:34.717157 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f722bf59\"" Sep 11 14:53:34.717319 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:34.717298 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f722bf59-dockercfg-65tmg\"" Sep 11 14:53:34.725383 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:34.725357 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f722bf59-79c869485-hdk7v"] Sep 11 14:53:34.733654 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:34.733627 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sqrb\" (UniqueName: \"kubernetes.io/projected/298d8799-0a14-4ce0-97b0-15ae6c4e54df-kube-api-access-2sqrb\") pod \"payload-processing-e2e-ait-f722bf59-79c869485-hdk7v\" (UID: \"298d8799-0a14-4ce0-97b0-15ae6c4e54df\") " pod="openshift-ingress/payload-processing-e2e-ait-f722bf59-79c869485-hdk7v" Sep 11 14:53:34.733782 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:34.733671 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/298d8799-0a14-4ce0-97b0-15ae6c4e54df-plugins-config-volume\") pod \"payload-processing-e2e-ait-f722bf59-79c869485-hdk7v\" (UID: \"298d8799-0a14-4ce0-97b0-15ae6c4e54df\") " pod="openshift-ingress/payload-processing-e2e-ait-f722bf59-79c869485-hdk7v" Sep 11 14:53:34.834396 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:34.834361 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sqrb\" (UniqueName: \"kubernetes.io/projected/298d8799-0a14-4ce0-97b0-15ae6c4e54df-kube-api-access-2sqrb\") pod \"payload-processing-e2e-ait-f722bf59-79c869485-hdk7v\" (UID: \"298d8799-0a14-4ce0-97b0-15ae6c4e54df\") " pod="openshift-ingress/payload-processing-e2e-ait-f722bf59-79c869485-hdk7v" Sep 11 14:53:34.834396 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:34.834396 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/298d8799-0a14-4ce0-97b0-15ae6c4e54df-plugins-config-volume\") pod \"payload-processing-e2e-ait-f722bf59-79c869485-hdk7v\" (UID: \"298d8799-0a14-4ce0-97b0-15ae6c4e54df\") " pod="openshift-ingress/payload-processing-e2e-ait-f722bf59-79c869485-hdk7v" Sep 11 14:53:34.835260 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:34.835233 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/298d8799-0a14-4ce0-97b0-15ae6c4e54df-plugins-config-volume\") pod \"payload-processing-e2e-ait-f722bf59-79c869485-hdk7v\" (UID: \"298d8799-0a14-4ce0-97b0-15ae6c4e54df\") " pod="openshift-ingress/payload-processing-e2e-ait-f722bf59-79c869485-hdk7v" Sep 11 14:53:34.843322 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:34.843292 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sqrb\" (UniqueName: \"kubernetes.io/projected/298d8799-0a14-4ce0-97b0-15ae6c4e54df-kube-api-access-2sqrb\") pod \"payload-processing-e2e-ait-f722bf59-79c869485-hdk7v\" (UID: \"298d8799-0a14-4ce0-97b0-15ae6c4e54df\") " pod="openshift-ingress/payload-processing-e2e-ait-f722bf59-79c869485-hdk7v" Sep 11 14:53:35.027869 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:35.027784 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f722bf59-79c869485-hdk7v" Sep 11 14:53:35.320435 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:35.320391 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f722bf59-79c869485-hdk7v"] Sep 11 14:53:36.211958 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:36.211877 2583 generic.go:356] "Generic (PLEG): container finished" podID="91ab0283-f94d-45bb-9b61-ad4d927d8b68" containerID="315a9c33c41e8e74ae348e51248b8be0e053a6fd86fc945fd0f45cd81ebe646f" exitCode=0 Sep 11 14:53:36.212411 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:36.211951 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" event={"ID":"91ab0283-f94d-45bb-9b61-ad4d927d8b68","Type":"ContainerDied","Data":"315a9c33c41e8e74ae348e51248b8be0e053a6fd86fc945fd0f45cd81ebe646f"} Sep 11 14:53:36.212485 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:36.212406 2583 scope.go:117] "RemoveContainer" containerID="315a9c33c41e8e74ae348e51248b8be0e053a6fd86fc945fd0f45cd81ebe646f" Sep 11 14:53:36.213638 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:36.213599 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:53:36.213638 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:36.213623 2583 generic.go:356] "Generic (PLEG): container finished" podID="9f745a9f-d6ba-4629-b8c0-8faacfbd4845" containerID="27a4bd6866cdf80cc2ce89ec8dda169627b693d5a2e4e5292cbe4b9c49628f6d" exitCode=0 Sep 11 14:53:36.213863 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:36.213685 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:53:36.213863 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:36.213694 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" event={"ID":"9f745a9f-d6ba-4629-b8c0-8faacfbd4845","Type":"ContainerDied","Data":"27a4bd6866cdf80cc2ce89ec8dda169627b693d5a2e4e5292cbe4b9c49628f6d"} Sep 11 14:53:36.213863 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:36.213766 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:53:36.214084 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:36.214067 2583 scope.go:117] "RemoveContainer" containerID="27a4bd6866cdf80cc2ce89ec8dda169627b693d5a2e4e5292cbe4b9c49628f6d" Sep 11 14:53:36.215171 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:36.215143 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:53:36.215255 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:36.215232 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:53:36.215305 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:36.215272 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:53:36.215305 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:36.215278 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f722bf59-79c869485-hdk7v" event={"ID":"298d8799-0a14-4ce0-97b0-15ae6c4e54df","Type":"ContainerStarted","Data":"70d5810f5ec5861f8e79e7f3f70ffa7bf17c150ab5151b8d53237bb2b3f49c2a"} Sep 11 14:53:36.215379 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:36.215318 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f722bf59-79c869485-hdk7v" event={"ID":"298d8799-0a14-4ce0-97b0-15ae6c4e54df","Type":"ContainerStarted","Data":"d0bc82885d548e0d62c4134b97ac7c3ee0a09af8583bc9380192035c49f4ab17"} Sep 11 14:53:36.215379 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:36.215364 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f722bf59-79c869485-hdk7v" Sep 11 14:53:36.266943 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:36.266880 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f722bf59-79c869485-hdk7v" podStartSLOduration=2.266464385 podStartE2EDuration="2.266464385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:36.264000395 +0000 UTC m=+807.507986082" watchObservedRunningTime="2026-09-11 14:53:36.266464385 +0000 UTC m=+807.510450045" Sep 11 14:53:37.225308 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:37.224325 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" event={"ID":"9f745a9f-d6ba-4629-b8c0-8faacfbd4845","Type":"ContainerStarted","Data":"4b3f3873c88ccdaa9cf8949c82ef5165f90a9536d9438b5791668876e75c6e30"} Sep 11 14:53:37.228646 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:37.228549 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" event={"ID":"91ab0283-f94d-45bb-9b61-ad4d927d8b68","Type":"ContainerStarted","Data":"06b129d54bd4b2ca7ca60c9f0d5c4ddaf1d4b6e90fb3368e14e6844c099934f4"} Sep 11 14:53:37.380870 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:37.380836 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:37.381035 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:37.381006 2583 patch_prober.go:28] interesting pod/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 11 14:53:37.381078 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:37.381058 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" podUID="91ab0283-f94d-45bb-9b61-ad4d927d8b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 11 14:53:38.196297 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:38.196257 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:38.196506 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:38.196386 2583 patch_prober.go:28] interesting pod/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Sep 11 14:53:38.196506 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:38.196430 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" podUID="9f745a9f-d6ba-4629-b8c0-8faacfbd4845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Sep 11 14:53:38.380831 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:38.380791 2583 patch_prober.go:28] interesting pod/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 11 14:53:38.381438 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:38.381406 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" podUID="91ab0283-f94d-45bb-9b61-ad4d927d8b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 11 14:53:39.119894 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:39.119859 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc" Sep 11 14:53:39.120178 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:39.120158 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj" Sep 11 14:53:39.202727 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:39.196240 2583 patch_prober.go:28] interesting pod/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Sep 11 14:53:39.202727 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:39.196311 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" podUID="9f745a9f-d6ba-4629-b8c0-8faacfbd4845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Sep 11 14:53:39.388257 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:39.388023 2583 patch_prober.go:28] interesting pod/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 11 14:53:39.388257 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:39.388098 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" podUID="91ab0283-f94d-45bb-9b61-ad4d927d8b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 11 14:53:40.196239 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:40.196193 2583 patch_prober.go:28] interesting pod/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Sep 11 14:53:40.196570 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:40.196532 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" podUID="9f745a9f-d6ba-4629-b8c0-8faacfbd4845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Sep 11 14:53:40.381301 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:40.381262 2583 patch_prober.go:28] interesting pod/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 11 14:53:40.381592 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:40.381544 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" podUID="91ab0283-f94d-45bb-9b61-ad4d927d8b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 11 14:53:41.196280 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:41.196234 2583 patch_prober.go:28] interesting pod/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Sep 11 14:53:41.196796 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:41.196303 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" podUID="9f745a9f-d6ba-4629-b8c0-8faacfbd4845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Sep 11 14:53:41.381277 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:41.381225 2583 patch_prober.go:28] interesting pod/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 11 14:53:41.381495 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:41.381304 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" podUID="91ab0283-f94d-45bb-9b61-ad4d927d8b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 11 14:53:41.963730 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:41.963573 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6"] Sep 11 14:53:42.007103 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.007068 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.010663 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.010638 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-90d6f5-openshift-default-dockercfg-jqfdm\"" Sep 11 14:53:42.012566 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.012534 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6"] Sep 11 14:53:42.120573 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.120542 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istiod-ca-cert\") pod \"e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.120768 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.120689 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istio-data\") pod \"e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.120768 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.120774 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-workload-certs\") pod \"e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.120931 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.120807 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8gfs\" (UniqueName: \"kubernetes.io/projected/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-kube-api-access-m8gfs\") pod \"e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.120931 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.120832 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istio-podinfo\") pod \"e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.120931 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.120864 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istio-envoy\") pod \"e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.121083 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.120946 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-credential-socket\") pod \"e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.121083 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.120983 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istio-token\") pod \"e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.121083 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.121022 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-workload-socket\") pod \"e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.196971 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.196931 2583 patch_prober.go:28] interesting pod/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Sep 11 14:53:42.197368 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.196987 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" podUID="9f745a9f-d6ba-4629-b8c0-8faacfbd4845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Sep 11 14:53:42.233310 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.233216 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-credential-socket\") pod \"e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.233310 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.233269 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istio-token\") pod \"e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.233522 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.233311 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-workload-socket\") pod \"e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.233522 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.233347 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istiod-ca-cert\") pod \"e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.233522 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.233407 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istio-data\") pod \"e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.233522 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.233433 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-workload-certs\") pod \"e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.233522 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.233465 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8gfs\" (UniqueName: \"kubernetes.io/projected/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-kube-api-access-m8gfs\") pod \"e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.233522 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.233492 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istio-podinfo\") pod \"e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.233848 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.233529 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istio-envoy\") pod \"e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.233848 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.233720 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-credential-socket\") pod \"e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.233848 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.233757 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-workload-socket\") pod \"e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.234009 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.233976 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-workload-certs\") pod \"e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.234731 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.234307 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istio-data\") pod \"e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.234731 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.234720 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istiod-ca-cert\") pod \"e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.236457 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.236417 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istio-envoy\") pod \"e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.238730 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.238232 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istio-podinfo\") pod \"e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.248723 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.248391 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8gfs\" (UniqueName: \"kubernetes.io/projected/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-kube-api-access-m8gfs\") pod \"e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.252722 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.250135 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istio-token\") pod \"e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.332205 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.332172 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:42.382421 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.381075 2583 patch_prober.go:28] interesting pod/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 11 14:53:42.382421 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.381154 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" podUID="91ab0283-f94d-45bb-9b61-ad4d927d8b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 11 14:53:42.679938 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.679893 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6"] Sep 11 14:53:42.693124 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.693082 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:53:42.693229 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.693154 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:53:42.693229 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:42.693182 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:53:43.196241 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:43.196199 2583 patch_prober.go:28] interesting pod/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Sep 11 14:53:43.196424 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:43.196279 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" podUID="9f745a9f-d6ba-4629-b8c0-8faacfbd4845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Sep 11 14:53:43.282246 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:43.282204 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" event={"ID":"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb","Type":"ContainerStarted","Data":"357ca16ae385a49a5476eea7e77bc433134a77dafee8a784a45a3e488de30e4e"} Sep 11 14:53:43.282787 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:43.282278 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" event={"ID":"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb","Type":"ContainerStarted","Data":"7436ea1252b16f602ad7074dd04778936388925a30ad3f614d6187cf29a3734f"} Sep 11 14:53:43.308668 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:43.308608 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" podStartSLOduration=2.308592204 podStartE2EDuration="2.308592204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:43.305433665 +0000 UTC m=+814.549419325" watchObservedRunningTime="2026-09-11 14:53:43.308592204 +0000 UTC m=+814.552577860" Sep 11 14:53:43.336257 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:43.336221 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:43.338362 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:43.338325 2583 patch_prober.go:28] interesting pod/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 11 14:53:43.338518 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:43.338393 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" podUID="3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 11 14:53:43.503396 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:43.503290 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:43.503842 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:43.503687 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:43.505741 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:43.505686 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 14:53:44.200788 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:44.200755 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:44.200975 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:44.200955 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq" Sep 11 14:53:44.201149 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:44.201130 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:44.201727 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:44.201696 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 14:53:44.338449 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:44.338419 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:45.296354 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:45.296323 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:45.297323 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:45.297302 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:53:46.304318 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:46.304283 2583 generic.go:356] "Generic (PLEG): container finished" podID="26130197-f0e3-4871-abb4-58a1f6331446" containerID="73fb9bb192d89744f9d7008d4565d56a8274e351d767bf4d8cd2f4d1a7e67216" exitCode=0 Sep 11 14:53:46.304789 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:46.304363 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" event={"ID":"26130197-f0e3-4871-abb4-58a1f6331446","Type":"ContainerDied","Data":"73fb9bb192d89744f9d7008d4565d56a8274e351d767bf4d8cd2f4d1a7e67216"} Sep 11 14:53:46.304916 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:46.304899 2583 scope.go:117] "RemoveContainer" containerID="73fb9bb192d89744f9d7008d4565d56a8274e351d767bf4d8cd2f4d1a7e67216" Sep 11 14:53:46.306170 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:46.306135 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:53:46.306278 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:46.306217 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:53:46.306278 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:46.306250 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:53:47.231606 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:47.231572 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f722bf59-79c869485-hdk7v" Sep 11 14:53:47.312562 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:47.312525 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" event={"ID":"26130197-f0e3-4871-abb4-58a1f6331446","Type":"ContainerStarted","Data":"e7eea1c2afa8f801df4efcf8c9fcc54a55e6251bcd41bff66b93c373ee9f4439"} Sep 11 14:53:47.868315 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:47.868280 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:47.868500 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:47.868457 2583 patch_prober.go:28] interesting pod/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 11 14:53:47.868555 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:47.868529 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" podUID="26130197-f0e3-4871-abb4-58a1f6331446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 11 14:53:48.868156 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:48.868120 2583 patch_prober.go:28] interesting pod/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 11 14:53:48.868549 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:48.868174 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" podUID="26130197-f0e3-4871-abb4-58a1f6331446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 11 14:53:49.868974 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:49.868933 2583 patch_prober.go:28] interesting pod/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 11 14:53:49.869356 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:49.868990 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" podUID="26130197-f0e3-4871-abb4-58a1f6331446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 11 14:53:50.794806 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:50.794771 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f722bf59-wzctz"] Sep 11 14:53:50.797172 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:50.797155 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f722bf59-wzctz" Sep 11 14:53:50.799487 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:50.799445 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nks8w\"" Sep 11 14:53:50.799642 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:50.799624 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:53:50.806942 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:50.806918 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f722bf59-wzctz"] Sep 11 14:53:50.868766 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:50.868697 2583 patch_prober.go:28] interesting pod/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 11 14:53:50.868955 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:50.868785 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" podUID="26130197-f0e3-4871-abb4-58a1f6331446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 11 14:53:50.937731 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:50.936231 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd583323-24cb-4851-bd33-c703e352ccc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f722bf59-wzctz\" (UID: \"dd583323-24cb-4851-bd33-c703e352ccc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f722bf59-wzctz" Sep 11 14:53:51.037791 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:51.037757 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd583323-24cb-4851-bd33-c703e352ccc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f722bf59-wzctz\" (UID: \"dd583323-24cb-4851-bd33-c703e352ccc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f722bf59-wzctz" Sep 11 14:53:51.038690 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:51.038637 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd583323-24cb-4851-bd33-c703e352ccc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f722bf59-wzctz\" (UID: \"dd583323-24cb-4851-bd33-c703e352ccc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f722bf59-wzctz" Sep 11 14:53:51.108604 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:51.108542 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f722bf59-wzctz" Sep 11 14:53:51.404203 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:51.404172 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f722bf59-wzctz"] Sep 11 14:53:51.405523 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:53:51.405499 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd583323_24cb_4851_bd33_c703e352ccc4.slice/crio-adb6b723201d1ee2be0a4ae6b27d7cb46d482885fb04f17dfdef207901710f90 WatchSource:0}: Error finding container adb6b723201d1ee2be0a4ae6b27d7cb46d482885fb04f17dfdef207901710f90: Status 404 returned error can't find the container with id adb6b723201d1ee2be0a4ae6b27d7cb46d482885fb04f17dfdef207901710f90 Sep 11 14:53:51.868778 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:51.868665 2583 patch_prober.go:28] interesting pod/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 11 14:53:51.868953 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:51.868795 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" podUID="26130197-f0e3-4871-abb4-58a1f6331446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 11 14:53:52.362149 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:52.362113 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f722bf59-wzctz" event={"ID":"dd583323-24cb-4851-bd33-c703e352ccc4","Type":"ContainerStarted","Data":"adb6b723201d1ee2be0a4ae6b27d7cb46d482885fb04f17dfdef207901710f90"} Sep 11 14:53:52.873157 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:52.873129 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:52.873781 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:52.873750 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:52.874693 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:52.874653 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:53:54.375920 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:54.375894 2583 generic.go:356] "Generic (PLEG): container finished" podID="dd583323-24cb-4851-bd33-c703e352ccc4" containerID="83326b5004e1793d554eaa7eb406fb3caa2fcc8dcca34e8ad8e0c5fce6c83364" exitCode=0 Sep 11 14:53:54.376247 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:54.375973 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f722bf59-wzctz" event={"ID":"dd583323-24cb-4851-bd33-c703e352ccc4","Type":"ContainerDied","Data":"83326b5004e1793d554eaa7eb406fb3caa2fcc8dcca34e8ad8e0c5fce6c83364"} Sep 11 14:53:55.514487 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:55.514459 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f722bf59-wzctz" Sep 11 14:53:55.690047 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:55.689979 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd583323-24cb-4851-bd33-c703e352ccc4-maas-api-service-ca\") pod \"dd583323-24cb-4851-bd33-c703e352ccc4\" (UID: \"dd583323-24cb-4851-bd33-c703e352ccc4\") " Sep 11 14:53:55.690353 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:55.690319 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd583323-24cb-4851-bd33-c703e352ccc4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dd583323-24cb-4851-bd33-c703e352ccc4" (UID: "dd583323-24cb-4851-bd33-c703e352ccc4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:53:55.791076 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:55.791052 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd583323-24cb-4851-bd33-c703e352ccc4-maas-api-service-ca\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:53:56.388314 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:56.388286 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f722bf59-wzctz" Sep 11 14:53:56.388314 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:56.388293 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f722bf59-wzctz" event={"ID":"dd583323-24cb-4851-bd33-c703e352ccc4","Type":"ContainerDied","Data":"adb6b723201d1ee2be0a4ae6b27d7cb46d482885fb04f17dfdef207901710f90"} Sep 11 14:53:56.388573 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:53:56.388339 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adb6b723201d1ee2be0a4ae6b27d7cb46d482885fb04f17dfdef207901710f90" Sep 11 14:54:01.412531 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:01.412488 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f722bf59-79c869485-hdk7v"] Sep 11 14:54:01.412979 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:01.412800 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f722bf59-79c869485-hdk7v" podUID="298d8799-0a14-4ce0-97b0-15ae6c4e54df" containerName="payload-processing" containerID="cri-o://70d5810f5ec5861f8e79e7f3f70ffa7bf17c150ab5151b8d53237bb2b3f49c2a" gracePeriod=30 Sep 11 14:54:01.588820 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:01.588785 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f722bf59-wzctz"] Sep 11 14:54:01.591308 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:01.591281 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f722bf59-wzctz"] Sep 11 14:54:01.706028 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:01.706005 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f722bf59-79c869485-hdk7v" Sep 11 14:54:01.759138 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:01.758244 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/298d8799-0a14-4ce0-97b0-15ae6c4e54df-plugins-config-volume\") pod \"298d8799-0a14-4ce0-97b0-15ae6c4e54df\" (UID: \"298d8799-0a14-4ce0-97b0-15ae6c4e54df\") " Sep 11 14:54:01.759138 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:01.758331 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sqrb\" (UniqueName: \"kubernetes.io/projected/298d8799-0a14-4ce0-97b0-15ae6c4e54df-kube-api-access-2sqrb\") pod \"298d8799-0a14-4ce0-97b0-15ae6c4e54df\" (UID: \"298d8799-0a14-4ce0-97b0-15ae6c4e54df\") " Sep 11 14:54:01.759138 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:01.759090 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/298d8799-0a14-4ce0-97b0-15ae6c4e54df-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "298d8799-0a14-4ce0-97b0-15ae6c4e54df" (UID: "298d8799-0a14-4ce0-97b0-15ae6c4e54df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:54:01.764519 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:01.764481 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298d8799-0a14-4ce0-97b0-15ae6c4e54df-kube-api-access-2sqrb" (OuterVolumeSpecName: "kube-api-access-2sqrb") pod "298d8799-0a14-4ce0-97b0-15ae6c4e54df" (UID: "298d8799-0a14-4ce0-97b0-15ae6c4e54df"). InnerVolumeSpecName "kube-api-access-2sqrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:54:01.859428 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:01.859391 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/298d8799-0a14-4ce0-97b0-15ae6c4e54df-plugins-config-volume\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:54:01.859428 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:01.859430 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sqrb\" (UniqueName: \"kubernetes.io/projected/298d8799-0a14-4ce0-97b0-15ae6c4e54df-kube-api-access-2sqrb\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:54:02.441855 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:02.441815 2583 generic.go:356] "Generic (PLEG): container finished" podID="298d8799-0a14-4ce0-97b0-15ae6c4e54df" containerID="70d5810f5ec5861f8e79e7f3f70ffa7bf17c150ab5151b8d53237bb2b3f49c2a" exitCode=0 Sep 11 14:54:02.442357 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:02.441960 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f722bf59-79c869485-hdk7v" event={"ID":"298d8799-0a14-4ce0-97b0-15ae6c4e54df","Type":"ContainerDied","Data":"70d5810f5ec5861f8e79e7f3f70ffa7bf17c150ab5151b8d53237bb2b3f49c2a"} Sep 11 14:54:02.442357 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:02.442014 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f722bf59-79c869485-hdk7v" event={"ID":"298d8799-0a14-4ce0-97b0-15ae6c4e54df","Type":"ContainerDied","Data":"d0bc82885d548e0d62c4134b97ac7c3ee0a09af8583bc9380192035c49f4ab17"} Sep 11 14:54:02.442357 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:02.442048 2583 scope.go:117] "RemoveContainer" containerID="70d5810f5ec5861f8e79e7f3f70ffa7bf17c150ab5151b8d53237bb2b3f49c2a" Sep 11 14:54:02.442357 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:02.442218 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f722bf59-79c869485-hdk7v" Sep 11 14:54:02.455810 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:02.455789 2583 scope.go:117] "RemoveContainer" containerID="70d5810f5ec5861f8e79e7f3f70ffa7bf17c150ab5151b8d53237bb2b3f49c2a" Sep 11 14:54:02.456223 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:54:02.456085 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70d5810f5ec5861f8e79e7f3f70ffa7bf17c150ab5151b8d53237bb2b3f49c2a\": container with ID starting with 70d5810f5ec5861f8e79e7f3f70ffa7bf17c150ab5151b8d53237bb2b3f49c2a not found: ID does not exist" containerID="70d5810f5ec5861f8e79e7f3f70ffa7bf17c150ab5151b8d53237bb2b3f49c2a" Sep 11 14:54:02.456223 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:02.456110 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d5810f5ec5861f8e79e7f3f70ffa7bf17c150ab5151b8d53237bb2b3f49c2a"} err="failed to get container status \"70d5810f5ec5861f8e79e7f3f70ffa7bf17c150ab5151b8d53237bb2b3f49c2a\": rpc error: code = NotFound desc = could not find container \"70d5810f5ec5861f8e79e7f3f70ffa7bf17c150ab5151b8d53237bb2b3f49c2a\": container with ID starting with 70d5810f5ec5861f8e79e7f3f70ffa7bf17c150ab5151b8d53237bb2b3f49c2a not found: ID does not exist" Sep 11 14:54:02.501088 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:02.501062 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f722bf59-79c869485-hdk7v"] Sep 11 14:54:02.506973 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:02.506950 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f722bf59-79c869485-hdk7v"] Sep 11 14:54:03.335826 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:03.335787 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="298d8799-0a14-4ce0-97b0-15ae6c4e54df" path="/var/lib/kubelet/pods/298d8799-0a14-4ce0-97b0-15ae6c4e54df/volumes" Sep 11 14:54:03.336365 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:03.336335 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd583323-24cb-4851-bd33-c703e352ccc4" path="/var/lib/kubelet/pods/dd583323-24cb-4851-bd33-c703e352ccc4/volumes" Sep 11 14:54:07.669585 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:07.669548 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k"] Sep 11 14:54:07.669971 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:07.669963 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="298d8799-0a14-4ce0-97b0-15ae6c4e54df" containerName="payload-processing" Sep 11 14:54:07.670010 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:07.669975 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="298d8799-0a14-4ce0-97b0-15ae6c4e54df" containerName="payload-processing" Sep 11 14:54:07.670010 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:07.669988 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd583323-24cb-4851-bd33-c703e352ccc4" containerName="revoke-keys" Sep 11 14:54:07.670010 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:07.669994 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd583323-24cb-4851-bd33-c703e352ccc4" containerName="revoke-keys" Sep 11 14:54:07.670103 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:07.670050 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="298d8799-0a14-4ce0-97b0-15ae6c4e54df" containerName="payload-processing" Sep 11 14:54:07.670103 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:07.670061 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd583323-24cb-4851-bd33-c703e352ccc4" containerName="revoke-keys" Sep 11 14:54:07.671903 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:07.671885 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k" Sep 11 14:54:07.674477 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:07.674457 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-4408ba1c-dockercfg-jp5jh\"" Sep 11 14:54:07.674750 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:07.674732 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-4408ba1c\"" Sep 11 14:54:07.685502 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:07.685477 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k"] Sep 11 14:54:07.712249 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:07.712220 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p4rv\" (UniqueName: \"kubernetes.io/projected/5964f072-418b-429a-a5c4-339fcf752e71-kube-api-access-2p4rv\") pod \"payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k\" (UID: \"5964f072-418b-429a-a5c4-339fcf752e71\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k" Sep 11 14:54:07.712391 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:07.712273 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5964f072-418b-429a-a5c4-339fcf752e71-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k\" (UID: \"5964f072-418b-429a-a5c4-339fcf752e71\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k" Sep 11 14:54:07.813805 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:07.813762 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p4rv\" (UniqueName: \"kubernetes.io/projected/5964f072-418b-429a-a5c4-339fcf752e71-kube-api-access-2p4rv\") pod \"payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k\" (UID: \"5964f072-418b-429a-a5c4-339fcf752e71\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k" Sep 11 14:54:07.813996 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:07.813854 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5964f072-418b-429a-a5c4-339fcf752e71-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k\" (UID: \"5964f072-418b-429a-a5c4-339fcf752e71\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k" Sep 11 14:54:07.814639 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:07.814613 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5964f072-418b-429a-a5c4-339fcf752e71-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k\" (UID: \"5964f072-418b-429a-a5c4-339fcf752e71\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k" Sep 11 14:54:07.822776 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:07.822753 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p4rv\" (UniqueName: \"kubernetes.io/projected/5964f072-418b-429a-a5c4-339fcf752e71-kube-api-access-2p4rv\") pod \"payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k\" (UID: \"5964f072-418b-429a-a5c4-339fcf752e71\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k" Sep 11 14:54:07.985033 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:07.984957 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k" Sep 11 14:54:08.258939 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:08.258915 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k"] Sep 11 14:54:08.260765 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:54:08.260733 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5964f072_418b_429a_a5c4_339fcf752e71.slice/crio-4c03ce03954123e4a5e80745e877e344dc3643139abca68aa519f11a2a94ade4 WatchSource:0}: Error finding container 4c03ce03954123e4a5e80745e877e344dc3643139abca68aa519f11a2a94ade4: Status 404 returned error can't find the container with id 4c03ce03954123e4a5e80745e877e344dc3643139abca68aa519f11a2a94ade4 Sep 11 14:54:08.484590 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:08.484562 2583 generic.go:356] "Generic (PLEG): container finished" podID="3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb" containerID="357ca16ae385a49a5476eea7e77bc433134a77dafee8a784a45a3e488de30e4e" exitCode=0 Sep 11 14:54:08.484762 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:08.484641 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" event={"ID":"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb","Type":"ContainerDied","Data":"357ca16ae385a49a5476eea7e77bc433134a77dafee8a784a45a3e488de30e4e"} Sep 11 14:54:08.485110 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:08.485090 2583 scope.go:117] "RemoveContainer" containerID="357ca16ae385a49a5476eea7e77bc433134a77dafee8a784a45a3e488de30e4e" Sep 11 14:54:08.486341 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:08.486310 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:54:08.486427 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:08.486384 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:54:08.486427 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:08.486418 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 14:54:08.486515 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:08.486477 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k" event={"ID":"5964f072-418b-429a-a5c4-339fcf752e71","Type":"ContainerStarted","Data":"a9970cf644dc8efb387325e90f8d66b432b32085d9166b590e89a24ac57d9cd0"} Sep 11 14:54:08.486515 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:08.486512 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k" event={"ID":"5964f072-418b-429a-a5c4-339fcf752e71","Type":"ContainerStarted","Data":"4c03ce03954123e4a5e80745e877e344dc3643139abca68aa519f11a2a94ade4"} Sep 11 14:54:08.486602 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:08.486552 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k" Sep 11 14:54:08.526629 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:08.526559 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k" podStartSLOduration=1.526544244 podStartE2EDuration="1.526544244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:54:08.522863635 +0000 UTC m=+839.766849294" watchObservedRunningTime="2026-09-11 14:54:08.526544244 +0000 UTC m=+839.770529897" Sep 11 14:54:09.501399 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:09.501290 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" event={"ID":"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb","Type":"ContainerStarted","Data":"a61a29e40ffc7fe6b414376c8c429f9a6757ae30a5033c2c5aacadff1e2e9281"} Sep 11 14:54:10.334396 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:10.334355 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:54:10.334611 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:10.334466 2583 patch_prober.go:28] interesting pod/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 11 14:54:10.334611 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:10.334511 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" podUID="3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 11 14:54:11.350729 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:11.349966 2583 patch_prober.go:28] interesting pod/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 11 14:54:11.350729 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:11.350024 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" podUID="3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 11 14:54:12.334464 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:12.334422 2583 patch_prober.go:28] interesting pod/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 11 14:54:12.334804 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:12.334775 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" podUID="3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 11 14:54:13.334376 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:13.334336 2583 patch_prober.go:28] interesting pod/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 11 14:54:13.334888 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:13.334401 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" podUID="3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 11 14:54:14.335121 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:14.335077 2583 patch_prober.go:28] interesting pod/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 11 14:54:14.335598 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:14.335145 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" podUID="3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 11 14:54:15.296432 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:15.296401 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:54:15.338873 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:15.338840 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:54:15.552295 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:15.552215 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 14:54:19.503793 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:19.503762 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k" Sep 11 14:54:30.624876 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:30.624836 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4408ba1c-nnclb"] Sep 11 14:54:30.632025 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:30.632003 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4408ba1c-nnclb" Sep 11 14:54:30.634559 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:30.634529 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:54:30.634769 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:30.634573 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nks8w\"" Sep 11 14:54:30.637685 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:30.637620 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4408ba1c-nnclb"] Sep 11 14:54:30.751424 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:30.751385 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1089b2ee-6a27-4029-ab43-2dfb04513307-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4408ba1c-nnclb\" (UID: \"1089b2ee-6a27-4029-ab43-2dfb04513307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4408ba1c-nnclb" Sep 11 14:54:30.852198 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:30.852152 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1089b2ee-6a27-4029-ab43-2dfb04513307-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4408ba1c-nnclb\" (UID: \"1089b2ee-6a27-4029-ab43-2dfb04513307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4408ba1c-nnclb" Sep 11 14:54:30.852809 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:30.852788 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1089b2ee-6a27-4029-ab43-2dfb04513307-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4408ba1c-nnclb\" (UID: \"1089b2ee-6a27-4029-ab43-2dfb04513307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4408ba1c-nnclb" Sep 11 14:54:30.945485 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:30.945453 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4408ba1c-nnclb" Sep 11 14:54:31.238532 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:31.238499 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4408ba1c-nnclb"] Sep 11 14:54:31.241108 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:54:31.241078 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1089b2ee_6a27_4029_ab43_2dfb04513307.slice/crio-8987c3be60da0c57033f20cdc458b7384805b3b13d66d37e88d9c7f515eaf820 WatchSource:0}: Error finding container 8987c3be60da0c57033f20cdc458b7384805b3b13d66d37e88d9c7f515eaf820: Status 404 returned error can't find the container with id 8987c3be60da0c57033f20cdc458b7384805b3b13d66d37e88d9c7f515eaf820 Sep 11 14:54:31.648236 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:31.648197 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4408ba1c-nnclb" event={"ID":"1089b2ee-6a27-4029-ab43-2dfb04513307","Type":"ContainerStarted","Data":"3a1b83748393cb85b8bd5928734e8112ff926851c7a24a6f788b1b9e541ded23"} Sep 11 14:54:31.648688 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:31.648254 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4408ba1c-nnclb" event={"ID":"1089b2ee-6a27-4029-ab43-2dfb04513307","Type":"ContainerStarted","Data":"8987c3be60da0c57033f20cdc458b7384805b3b13d66d37e88d9c7f515eaf820"} Sep 11 14:54:31.664336 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:31.664296 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4408ba1c-nnclb" podStartSLOduration=1.6642826080000002 podStartE2EDuration="1.664282608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:54:31.663269507 +0000 UTC m=+862.907255168" watchObservedRunningTime="2026-09-11 14:54:31.664282608 +0000 UTC m=+862.908268267" Sep 11 14:54:32.655642 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:32.655603 2583 generic.go:356] "Generic (PLEG): container finished" podID="1089b2ee-6a27-4029-ab43-2dfb04513307" containerID="3a1b83748393cb85b8bd5928734e8112ff926851c7a24a6f788b1b9e541ded23" exitCode=0 Sep 11 14:54:32.656059 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:32.655678 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4408ba1c-nnclb" event={"ID":"1089b2ee-6a27-4029-ab43-2dfb04513307","Type":"ContainerDied","Data":"3a1b83748393cb85b8bd5928734e8112ff926851c7a24a6f788b1b9e541ded23"} Sep 11 14:54:33.801421 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:33.801397 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4408ba1c-nnclb" Sep 11 14:54:33.984464 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:33.984379 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1089b2ee-6a27-4029-ab43-2dfb04513307-maas-api-service-ca\") pod \"1089b2ee-6a27-4029-ab43-2dfb04513307\" (UID: \"1089b2ee-6a27-4029-ab43-2dfb04513307\") " Sep 11 14:54:33.984752 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:33.984730 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1089b2ee-6a27-4029-ab43-2dfb04513307-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1089b2ee-6a27-4029-ab43-2dfb04513307" (UID: "1089b2ee-6a27-4029-ab43-2dfb04513307"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:54:34.086161 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:34.086124 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1089b2ee-6a27-4029-ab43-2dfb04513307-maas-api-service-ca\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:54:34.667720 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:34.667680 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4408ba1c-nnclb" Sep 11 14:54:34.667886 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:34.667730 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4408ba1c-nnclb" event={"ID":"1089b2ee-6a27-4029-ab43-2dfb04513307","Type":"ContainerDied","Data":"8987c3be60da0c57033f20cdc458b7384805b3b13d66d37e88d9c7f515eaf820"} Sep 11 14:54:34.667886 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:34.667775 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8987c3be60da0c57033f20cdc458b7384805b3b13d66d37e88d9c7f515eaf820" Sep 11 14:54:36.950953 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:36.950914 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k"] Sep 11 14:54:36.951404 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:36.951173 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k" podUID="5964f072-418b-429a-a5c4-339fcf752e71" containerName="payload-pre-processing" containerID="cri-o://a9970cf644dc8efb387325e90f8d66b432b32085d9166b590e89a24ac57d9cd0" gracePeriod=30 Sep 11 14:54:37.130437 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:37.130400 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4408ba1c-nnclb"] Sep 11 14:54:37.133136 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:37.133111 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4408ba1c-nnclb"] Sep 11 14:54:37.192148 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:37.192124 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k" Sep 11 14:54:37.318905 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:37.318810 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2p4rv\" (UniqueName: \"kubernetes.io/projected/5964f072-418b-429a-a5c4-339fcf752e71-kube-api-access-2p4rv\") pod \"5964f072-418b-429a-a5c4-339fcf752e71\" (UID: \"5964f072-418b-429a-a5c4-339fcf752e71\") " Sep 11 14:54:37.319057 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:37.318905 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5964f072-418b-429a-a5c4-339fcf752e71-plugins-config-volume\") pod \"5964f072-418b-429a-a5c4-339fcf752e71\" (UID: \"5964f072-418b-429a-a5c4-339fcf752e71\") " Sep 11 14:54:37.319271 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:37.319236 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5964f072-418b-429a-a5c4-339fcf752e71-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5964f072-418b-429a-a5c4-339fcf752e71" (UID: "5964f072-418b-429a-a5c4-339fcf752e71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:54:37.320862 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:37.320842 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5964f072-418b-429a-a5c4-339fcf752e71-kube-api-access-2p4rv" (OuterVolumeSpecName: "kube-api-access-2p4rv") pod "5964f072-418b-429a-a5c4-339fcf752e71" (UID: "5964f072-418b-429a-a5c4-339fcf752e71"). InnerVolumeSpecName "kube-api-access-2p4rv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:54:37.330820 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:37.330798 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1089b2ee-6a27-4029-ab43-2dfb04513307" path="/var/lib/kubelet/pods/1089b2ee-6a27-4029-ab43-2dfb04513307/volumes" Sep 11 14:54:37.419822 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:37.419783 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p4rv\" (UniqueName: \"kubernetes.io/projected/5964f072-418b-429a-a5c4-339fcf752e71-kube-api-access-2p4rv\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:54:37.419822 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:37.419813 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5964f072-418b-429a-a5c4-339fcf752e71-plugins-config-volume\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:54:37.686979 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:37.686943 2583 generic.go:356] "Generic (PLEG): container finished" podID="5964f072-418b-429a-a5c4-339fcf752e71" containerID="a9970cf644dc8efb387325e90f8d66b432b32085d9166b590e89a24ac57d9cd0" exitCode=0 Sep 11 14:54:37.687162 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:37.686993 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k" Sep 11 14:54:37.687162 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:37.687025 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k" event={"ID":"5964f072-418b-429a-a5c4-339fcf752e71","Type":"ContainerDied","Data":"a9970cf644dc8efb387325e90f8d66b432b32085d9166b590e89a24ac57d9cd0"} Sep 11 14:54:37.687162 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:37.687086 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k" event={"ID":"5964f072-418b-429a-a5c4-339fcf752e71","Type":"ContainerDied","Data":"4c03ce03954123e4a5e80745e877e344dc3643139abca68aa519f11a2a94ade4"} Sep 11 14:54:37.687162 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:37.687112 2583 scope.go:117] "RemoveContainer" containerID="a9970cf644dc8efb387325e90f8d66b432b32085d9166b590e89a24ac57d9cd0" Sep 11 14:54:37.696153 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:37.696135 2583 scope.go:117] "RemoveContainer" containerID="a9970cf644dc8efb387325e90f8d66b432b32085d9166b590e89a24ac57d9cd0" Sep 11 14:54:37.696415 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:54:37.696396 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9970cf644dc8efb387325e90f8d66b432b32085d9166b590e89a24ac57d9cd0\": container with ID starting with a9970cf644dc8efb387325e90f8d66b432b32085d9166b590e89a24ac57d9cd0 not found: ID does not exist" containerID="a9970cf644dc8efb387325e90f8d66b432b32085d9166b590e89a24ac57d9cd0" Sep 11 14:54:37.696483 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:37.696426 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9970cf644dc8efb387325e90f8d66b432b32085d9166b590e89a24ac57d9cd0"} err="failed to get container status \"a9970cf644dc8efb387325e90f8d66b432b32085d9166b590e89a24ac57d9cd0\": rpc error: code = NotFound desc = could not find container \"a9970cf644dc8efb387325e90f8d66b432b32085d9166b590e89a24ac57d9cd0\": container with ID starting with a9970cf644dc8efb387325e90f8d66b432b32085d9166b590e89a24ac57d9cd0 not found: ID does not exist" Sep 11 14:54:37.703886 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:37.703864 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k"] Sep 11 14:54:37.708487 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:37.708466 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4408ba1c-b9bd4d94-jnc4k"] Sep 11 14:54:39.330981 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:39.330951 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5964f072-418b-429a-a5c4-339fcf752e71" path="/var/lib/kubelet/pods/5964f072-418b-429a-a5c4-339fcf752e71/volumes" Sep 11 14:54:43.440964 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:43.440914 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575"] Sep 11 14:54:43.441909 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:43.441882 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5964f072-418b-429a-a5c4-339fcf752e71" containerName="payload-pre-processing" Sep 11 14:54:43.441909 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:43.441910 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5964f072-418b-429a-a5c4-339fcf752e71" containerName="payload-pre-processing" Sep 11 14:54:43.442075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:43.441936 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1089b2ee-6a27-4029-ab43-2dfb04513307" containerName="revoke-keys" Sep 11 14:54:43.442075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:43.441945 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1089b2ee-6a27-4029-ab43-2dfb04513307" containerName="revoke-keys" Sep 11 14:54:43.442193 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:43.442106 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5964f072-418b-429a-a5c4-339fcf752e71" containerName="payload-pre-processing" Sep 11 14:54:43.442193 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:43.442129 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1089b2ee-6a27-4029-ab43-2dfb04513307" containerName="revoke-keys" Sep 11 14:54:43.447444 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:43.447423 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575" Sep 11 14:54:43.450071 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:43.450037 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c0b7dbbd\"" Sep 11 14:54:43.450165 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:43.450074 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c0b7dbbd-dockercfg-dbpm8\"" Sep 11 14:54:43.453936 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:43.453914 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575"] Sep 11 14:54:43.476481 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:43.476453 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b3dcbb-801c-4223-a248-20d653d45100-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575\" (UID: \"f3b3dcbb-801c-4223-a248-20d653d45100\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575" Sep 11 14:54:43.476593 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:43.476508 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6sfl\" (UniqueName: \"kubernetes.io/projected/f3b3dcbb-801c-4223-a248-20d653d45100-kube-api-access-g6sfl\") pod \"payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575\" (UID: \"f3b3dcbb-801c-4223-a248-20d653d45100\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575" Sep 11 14:54:43.578174 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:43.578136 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b3dcbb-801c-4223-a248-20d653d45100-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575\" (UID: \"f3b3dcbb-801c-4223-a248-20d653d45100\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575" Sep 11 14:54:43.578343 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:43.578199 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6sfl\" (UniqueName: \"kubernetes.io/projected/f3b3dcbb-801c-4223-a248-20d653d45100-kube-api-access-g6sfl\") pod \"payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575\" (UID: \"f3b3dcbb-801c-4223-a248-20d653d45100\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575" Sep 11 14:54:43.578894 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:43.578874 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b3dcbb-801c-4223-a248-20d653d45100-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575\" (UID: \"f3b3dcbb-801c-4223-a248-20d653d45100\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575" Sep 11 14:54:43.586585 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:43.586554 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6sfl\" (UniqueName: \"kubernetes.io/projected/f3b3dcbb-801c-4223-a248-20d653d45100-kube-api-access-g6sfl\") pod \"payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575\" (UID: \"f3b3dcbb-801c-4223-a248-20d653d45100\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575" Sep 11 14:54:43.759156 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:43.759068 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575" Sep 11 14:54:44.043008 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:44.042977 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575"] Sep 11 14:54:44.044539 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:54:44.044512 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b3dcbb_801c_4223_a248_20d653d45100.slice/crio-13e3379098cbe7bfa77b894eb2040d2fc44d94075ae05e59f61ddd8746e82642 WatchSource:0}: Error finding container 13e3379098cbe7bfa77b894eb2040d2fc44d94075ae05e59f61ddd8746e82642: Status 404 returned error can't find the container with id 13e3379098cbe7bfa77b894eb2040d2fc44d94075ae05e59f61ddd8746e82642 Sep 11 14:54:44.737089 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:44.737054 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575" event={"ID":"f3b3dcbb-801c-4223-a248-20d653d45100","Type":"ContainerStarted","Data":"1fd64c405cb6898ecebbdfa82299a4788189d8cf50a37541fb11ccb850de5bc0"} Sep 11 14:54:44.737504 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:44.737102 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575" event={"ID":"f3b3dcbb-801c-4223-a248-20d653d45100","Type":"ContainerStarted","Data":"13e3379098cbe7bfa77b894eb2040d2fc44d94075ae05e59f61ddd8746e82642"} Sep 11 14:54:44.737504 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:44.737254 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575" Sep 11 14:54:44.756260 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:44.756207 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575" podStartSLOduration=1.7561909930000001 podStartE2EDuration="1.756190993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:54:44.754140982 +0000 UTC m=+875.998126642" watchObservedRunningTime="2026-09-11 14:54:44.756190993 +0000 UTC m=+876.000176652" Sep 11 14:54:55.744460 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:54:55.744427 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575" Sep 11 14:55:06.772402 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:06.772327 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7dbbd-tfpz8"] Sep 11 14:55:06.776120 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:06.776100 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7dbbd-tfpz8" Sep 11 14:55:06.778501 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:06.778483 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:55:06.778573 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:06.778498 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nks8w\"" Sep 11 14:55:06.786631 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:06.786604 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7dbbd-tfpz8"] Sep 11 14:55:06.891140 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:06.891115 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af210356-ddbc-4925-95c8-b104cc1d130d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c0b7dbbd-tfpz8\" (UID: \"af210356-ddbc-4925-95c8-b104cc1d130d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7dbbd-tfpz8" Sep 11 14:55:06.992365 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:06.992339 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af210356-ddbc-4925-95c8-b104cc1d130d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c0b7dbbd-tfpz8\" (UID: \"af210356-ddbc-4925-95c8-b104cc1d130d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7dbbd-tfpz8" Sep 11 14:55:06.993014 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:06.992998 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af210356-ddbc-4925-95c8-b104cc1d130d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c0b7dbbd-tfpz8\" (UID: \"af210356-ddbc-4925-95c8-b104cc1d130d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7dbbd-tfpz8" Sep 11 14:55:07.112638 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:07.112572 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7dbbd-tfpz8" Sep 11 14:55:07.385394 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:07.385369 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7dbbd-tfpz8"] Sep 11 14:55:07.386966 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:55:07.386940 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf210356_ddbc_4925_95c8_b104cc1d130d.slice/crio-2872757f1408ce4c8e499923f2a2acb96ef3c080f86ae490b12f6235f67ea680 WatchSource:0}: Error finding container 2872757f1408ce4c8e499923f2a2acb96ef3c080f86ae490b12f6235f67ea680: Status 404 returned error can't find the container with id 2872757f1408ce4c8e499923f2a2acb96ef3c080f86ae490b12f6235f67ea680 Sep 11 14:55:07.880948 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:07.880918 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7dbbd-tfpz8" event={"ID":"af210356-ddbc-4925-95c8-b104cc1d130d","Type":"ContainerStarted","Data":"30c16b4771b9369ff492ae512b41395552c714a9d51ae70d52ecad84b3a8c6be"} Sep 11 14:55:07.881304 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:07.880968 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7dbbd-tfpz8" event={"ID":"af210356-ddbc-4925-95c8-b104cc1d130d","Type":"ContainerStarted","Data":"2872757f1408ce4c8e499923f2a2acb96ef3c080f86ae490b12f6235f67ea680"} Sep 11 14:55:07.900350 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:07.900303 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7dbbd-tfpz8" podStartSLOduration=1.900289766 podStartE2EDuration="1.900289766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:55:07.898594855 +0000 UTC m=+899.142580515" watchObservedRunningTime="2026-09-11 14:55:07.900289766 +0000 UTC m=+899.144275424" Sep 11 14:55:08.891363 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:08.891324 2583 generic.go:356] "Generic (PLEG): container finished" podID="af210356-ddbc-4925-95c8-b104cc1d130d" containerID="30c16b4771b9369ff492ae512b41395552c714a9d51ae70d52ecad84b3a8c6be" exitCode=0 Sep 11 14:55:08.891782 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:08.891390 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7dbbd-tfpz8" event={"ID":"af210356-ddbc-4925-95c8-b104cc1d130d","Type":"ContainerDied","Data":"30c16b4771b9369ff492ae512b41395552c714a9d51ae70d52ecad84b3a8c6be"} Sep 11 14:55:09.216850 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:09.216756 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lwrt2_57d4ba26-384d-42f8-86a1-335e27cdf762/console-operator/1.log" Sep 11 14:55:09.218028 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:09.218004 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lwrt2_57d4ba26-384d-42f8-86a1-335e27cdf762/console-operator/1.log" Sep 11 14:55:10.028361 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:10.028338 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7dbbd-tfpz8" Sep 11 14:55:10.122001 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:10.121970 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af210356-ddbc-4925-95c8-b104cc1d130d-maas-api-service-ca\") pod \"af210356-ddbc-4925-95c8-b104cc1d130d\" (UID: \"af210356-ddbc-4925-95c8-b104cc1d130d\") " Sep 11 14:55:10.122281 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:10.122260 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af210356-ddbc-4925-95c8-b104cc1d130d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "af210356-ddbc-4925-95c8-b104cc1d130d" (UID: "af210356-ddbc-4925-95c8-b104cc1d130d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:55:10.222959 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:10.222885 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af210356-ddbc-4925-95c8-b104cc1d130d-maas-api-service-ca\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:55:10.904682 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:10.904651 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7dbbd-tfpz8" Sep 11 14:55:10.904870 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:10.904681 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7dbbd-tfpz8" event={"ID":"af210356-ddbc-4925-95c8-b104cc1d130d","Type":"ContainerDied","Data":"2872757f1408ce4c8e499923f2a2acb96ef3c080f86ae490b12f6235f67ea680"} Sep 11 14:55:10.904870 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:10.904746 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2872757f1408ce4c8e499923f2a2acb96ef3c080f86ae490b12f6235f67ea680" Sep 11 14:55:12.498142 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:12.498111 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575"] Sep 11 14:55:12.498616 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:12.498346 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575" podUID="f3b3dcbb-801c-4223-a248-20d653d45100" containerName="payload-pre-processing" containerID="cri-o://1fd64c405cb6898ecebbdfa82299a4788189d8cf50a37541fb11ccb850de5bc0" gracePeriod=30 Sep 11 14:55:12.668742 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:12.668664 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7dbbd-tfpz8"] Sep 11 14:55:12.674045 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:12.674019 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7dbbd-tfpz8"] Sep 11 14:55:12.746682 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:12.746654 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575" Sep 11 14:55:12.851350 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:12.851218 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b3dcbb-801c-4223-a248-20d653d45100-plugins-config-volume\") pod \"f3b3dcbb-801c-4223-a248-20d653d45100\" (UID: \"f3b3dcbb-801c-4223-a248-20d653d45100\") " Sep 11 14:55:12.851529 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:12.851414 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6sfl\" (UniqueName: \"kubernetes.io/projected/f3b3dcbb-801c-4223-a248-20d653d45100-kube-api-access-g6sfl\") pod \"f3b3dcbb-801c-4223-a248-20d653d45100\" (UID: \"f3b3dcbb-801c-4223-a248-20d653d45100\") " Sep 11 14:55:12.851597 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:12.851580 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b3dcbb-801c-4223-a248-20d653d45100-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f3b3dcbb-801c-4223-a248-20d653d45100" (UID: "f3b3dcbb-801c-4223-a248-20d653d45100"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:55:12.851762 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:12.851745 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b3dcbb-801c-4223-a248-20d653d45100-plugins-config-volume\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:55:12.853459 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:12.853431 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b3dcbb-801c-4223-a248-20d653d45100-kube-api-access-g6sfl" (OuterVolumeSpecName: "kube-api-access-g6sfl") pod "f3b3dcbb-801c-4223-a248-20d653d45100" (UID: "f3b3dcbb-801c-4223-a248-20d653d45100"). InnerVolumeSpecName "kube-api-access-g6sfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:55:12.921348 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:12.921320 2583 generic.go:356] "Generic (PLEG): container finished" podID="f3b3dcbb-801c-4223-a248-20d653d45100" containerID="1fd64c405cb6898ecebbdfa82299a4788189d8cf50a37541fb11ccb850de5bc0" exitCode=0 Sep 11 14:55:12.921496 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:12.921361 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575" Sep 11 14:55:12.921496 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:12.921401 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575" event={"ID":"f3b3dcbb-801c-4223-a248-20d653d45100","Type":"ContainerDied","Data":"1fd64c405cb6898ecebbdfa82299a4788189d8cf50a37541fb11ccb850de5bc0"} Sep 11 14:55:12.921496 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:12.921460 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575" event={"ID":"f3b3dcbb-801c-4223-a248-20d653d45100","Type":"ContainerDied","Data":"13e3379098cbe7bfa77b894eb2040d2fc44d94075ae05e59f61ddd8746e82642"} Sep 11 14:55:12.921496 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:12.921489 2583 scope.go:117] "RemoveContainer" containerID="1fd64c405cb6898ecebbdfa82299a4788189d8cf50a37541fb11ccb850de5bc0" Sep 11 14:55:12.931038 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:12.931026 2583 scope.go:117] "RemoveContainer" containerID="1fd64c405cb6898ecebbdfa82299a4788189d8cf50a37541fb11ccb850de5bc0" Sep 11 14:55:12.931285 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:55:12.931265 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fd64c405cb6898ecebbdfa82299a4788189d8cf50a37541fb11ccb850de5bc0\": container with ID starting with 1fd64c405cb6898ecebbdfa82299a4788189d8cf50a37541fb11ccb850de5bc0 not found: ID does not exist" containerID="1fd64c405cb6898ecebbdfa82299a4788189d8cf50a37541fb11ccb850de5bc0" Sep 11 14:55:12.931332 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:12.931294 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd64c405cb6898ecebbdfa82299a4788189d8cf50a37541fb11ccb850de5bc0"} err="failed to get container status \"1fd64c405cb6898ecebbdfa82299a4788189d8cf50a37541fb11ccb850de5bc0\": rpc error: code = NotFound desc = could not find container \"1fd64c405cb6898ecebbdfa82299a4788189d8cf50a37541fb11ccb850de5bc0\": container with ID starting with 1fd64c405cb6898ecebbdfa82299a4788189d8cf50a37541fb11ccb850de5bc0 not found: ID does not exist" Sep 11 14:55:12.943313 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:12.943288 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575"] Sep 11 14:55:12.947644 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:12.947625 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c0b7dbbd-666c445f58-9b575"] Sep 11 14:55:12.952446 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:12.952428 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6sfl\" (UniqueName: \"kubernetes.io/projected/f3b3dcbb-801c-4223-a248-20d653d45100-kube-api-access-g6sfl\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:55:13.330936 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:13.330903 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af210356-ddbc-4925-95c8-b104cc1d130d" path="/var/lib/kubelet/pods/af210356-ddbc-4925-95c8-b104cc1d130d/volumes" Sep 11 14:55:13.331287 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:13.331274 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3b3dcbb-801c-4223-a248-20d653d45100" path="/var/lib/kubelet/pods/f3b3dcbb-801c-4223-a248-20d653d45100/volumes" Sep 11 14:55:22.823191 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:22.823153 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx"] Sep 11 14:55:22.823660 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:22.823547 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af210356-ddbc-4925-95c8-b104cc1d130d" containerName="revoke-keys" Sep 11 14:55:22.823660 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:22.823557 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="af210356-ddbc-4925-95c8-b104cc1d130d" containerName="revoke-keys" Sep 11 14:55:22.823660 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:22.823578 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3b3dcbb-801c-4223-a248-20d653d45100" containerName="payload-pre-processing" Sep 11 14:55:22.823660 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:22.823584 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3b3dcbb-801c-4223-a248-20d653d45100" containerName="payload-pre-processing" Sep 11 14:55:22.823660 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:22.823641 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="af210356-ddbc-4925-95c8-b104cc1d130d" containerName="revoke-keys" Sep 11 14:55:22.823660 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:22.823653 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3b3dcbb-801c-4223-a248-20d653d45100" containerName="payload-pre-processing" Sep 11 14:55:22.828170 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:22.828143 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx" Sep 11 14:55:22.830552 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:22.830520 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-58b751e1\"" Sep 11 14:55:22.831056 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:22.831034 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l"] Sep 11 14:55:22.834431 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:22.834407 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l" Sep 11 14:55:22.839924 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:22.839895 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx"] Sep 11 14:55:22.849520 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:22.849498 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e11d75-135f-483c-9ee3-c5c09c1df857-plugins-config-volume\") pod \"payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l\" (UID: \"e7e11d75-135f-483c-9ee3-c5c09c1df857\") " pod="openshift-ingress/payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l" Sep 11 14:55:22.849724 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:22.849683 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9rmw\" (UniqueName: \"kubernetes.io/projected/e7e11d75-135f-483c-9ee3-c5c09c1df857-kube-api-access-v9rmw\") pod \"payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l\" (UID: \"e7e11d75-135f-483c-9ee3-c5c09c1df857\") " pod="openshift-ingress/payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l" Sep 11 14:55:22.849845 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:22.849746 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9403bce2-551c-4523-bfe8-f3f271700b13-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx\" (UID: \"9403bce2-551c-4523-bfe8-f3f271700b13\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx" Sep 11 14:55:22.849845 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:22.849770 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dnfg\" (UniqueName: \"kubernetes.io/projected/9403bce2-551c-4523-bfe8-f3f271700b13-kube-api-access-2dnfg\") pod \"payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx\" (UID: \"9403bce2-551c-4523-bfe8-f3f271700b13\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx" Sep 11 14:55:22.850813 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:22.850790 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l"] Sep 11 14:55:22.953166 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:22.950555 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e11d75-135f-483c-9ee3-c5c09c1df857-plugins-config-volume\") pod \"payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l\" (UID: \"e7e11d75-135f-483c-9ee3-c5c09c1df857\") " pod="openshift-ingress/payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l" Sep 11 14:55:22.953166 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:22.950622 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9rmw\" (UniqueName: \"kubernetes.io/projected/e7e11d75-135f-483c-9ee3-c5c09c1df857-kube-api-access-v9rmw\") pod \"payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l\" (UID: \"e7e11d75-135f-483c-9ee3-c5c09c1df857\") " pod="openshift-ingress/payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l" Sep 11 14:55:22.953166 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:22.950658 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9403bce2-551c-4523-bfe8-f3f271700b13-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx\" (UID: \"9403bce2-551c-4523-bfe8-f3f271700b13\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx" Sep 11 14:55:22.953166 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:22.950683 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dnfg\" (UniqueName: \"kubernetes.io/projected/9403bce2-551c-4523-bfe8-f3f271700b13-kube-api-access-2dnfg\") pod \"payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx\" (UID: \"9403bce2-551c-4523-bfe8-f3f271700b13\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx" Sep 11 14:55:22.953166 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:22.951948 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e11d75-135f-483c-9ee3-c5c09c1df857-plugins-config-volume\") pod \"payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l\" (UID: \"e7e11d75-135f-483c-9ee3-c5c09c1df857\") " pod="openshift-ingress/payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l" Sep 11 14:55:22.953166 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:22.953082 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9403bce2-551c-4523-bfe8-f3f271700b13-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx\" (UID: \"9403bce2-551c-4523-bfe8-f3f271700b13\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx" Sep 11 14:55:22.961782 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:22.961698 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dnfg\" (UniqueName: \"kubernetes.io/projected/9403bce2-551c-4523-bfe8-f3f271700b13-kube-api-access-2dnfg\") pod \"payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx\" (UID: \"9403bce2-551c-4523-bfe8-f3f271700b13\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx" Sep 11 14:55:22.966189 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:22.964092 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9rmw\" (UniqueName: \"kubernetes.io/projected/e7e11d75-135f-483c-9ee3-c5c09c1df857-kube-api-access-v9rmw\") pod \"payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l\" (UID: \"e7e11d75-135f-483c-9ee3-c5c09c1df857\") " pod="openshift-ingress/payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l" Sep 11 14:55:23.143494 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:23.143463 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx" Sep 11 14:55:23.156933 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:23.156911 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l" Sep 11 14:55:23.419777 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:23.419750 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx"] Sep 11 14:55:23.420683 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:55:23.420657 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9403bce2_551c_4523_bfe8_f3f271700b13.slice/crio-827f171a9de3cd4798f55a209a24b1ee5a70fcfa8a1a84fa9183e9a8d823c63b WatchSource:0}: Error finding container 827f171a9de3cd4798f55a209a24b1ee5a70fcfa8a1a84fa9183e9a8d823c63b: Status 404 returned error can't find the container with id 827f171a9de3cd4798f55a209a24b1ee5a70fcfa8a1a84fa9183e9a8d823c63b Sep 11 14:55:23.440541 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:23.440520 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l"] Sep 11 14:55:23.444485 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:55:23.444450 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e11d75_135f_483c_9ee3_c5c09c1df857.slice/crio-ad47f029380522aaf52bf050d3839b956f6eff0cf5f5ce7fe169b1d223d1bd40 WatchSource:0}: Error finding container ad47f029380522aaf52bf050d3839b956f6eff0cf5f5ce7fe169b1d223d1bd40: Status 404 returned error can't find the container with id ad47f029380522aaf52bf050d3839b956f6eff0cf5f5ce7fe169b1d223d1bd40 Sep 11 14:55:23.994776 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:23.994737 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l" event={"ID":"e7e11d75-135f-483c-9ee3-c5c09c1df857","Type":"ContainerStarted","Data":"c7e8dfff0454796c4391c0e80173e756dbb90e2282274d007387dcfb6f9b1245"} Sep 11 14:55:23.995216 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:23.994803 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l" event={"ID":"e7e11d75-135f-483c-9ee3-c5c09c1df857","Type":"ContainerStarted","Data":"ad47f029380522aaf52bf050d3839b956f6eff0cf5f5ce7fe169b1d223d1bd40"} Sep 11 14:55:23.995216 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:23.994942 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l" Sep 11 14:55:23.996366 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:23.996344 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx" event={"ID":"9403bce2-551c-4523-bfe8-f3f271700b13","Type":"ContainerStarted","Data":"8a65c9d9e4e59ef2b861a61637a57c3107a2c724fe0101b87e8b0b8e3d9835bb"} Sep 11 14:55:23.996485 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:23.996380 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx" event={"ID":"9403bce2-551c-4523-bfe8-f3f271700b13","Type":"ContainerStarted","Data":"827f171a9de3cd4798f55a209a24b1ee5a70fcfa8a1a84fa9183e9a8d823c63b"} Sep 11 14:55:23.996485 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:23.996470 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx" Sep 11 14:55:24.022714 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:24.022651 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l" podStartSLOduration=2.022633534 podStartE2EDuration="2.022633534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:55:24.020753239 +0000 UTC m=+915.264738899" watchObservedRunningTime="2026-09-11 14:55:24.022633534 +0000 UTC m=+915.266619196" Sep 11 14:55:24.046924 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:24.046884 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx" podStartSLOduration=2.046852305 podStartE2EDuration="2.046852305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:55:24.042186788 +0000 UTC m=+915.286172459" watchObservedRunningTime="2026-09-11 14:55:24.046852305 +0000 UTC m=+915.290837964" Sep 11 14:55:35.004121 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:35.004090 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx" Sep 11 14:55:35.004659 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:35.004161 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l" Sep 11 14:55:47.969592 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:47.969559 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58b751e1-n95vg"] Sep 11 14:55:47.973040 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:47.973021 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58b751e1-n95vg" Sep 11 14:55:47.975443 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:47.975418 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:55:47.975547 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:47.975531 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nks8w\"" Sep 11 14:55:47.981123 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:47.980858 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58b751e1-n95vg"] Sep 11 14:55:48.072679 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:48.072647 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b63c527c-4efd-4c9d-8665-0d40cffec787-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-58b751e1-n95vg\" (UID: \"b63c527c-4efd-4c9d-8665-0d40cffec787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58b751e1-n95vg" Sep 11 14:55:48.174150 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:48.174119 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b63c527c-4efd-4c9d-8665-0d40cffec787-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-58b751e1-n95vg\" (UID: \"b63c527c-4efd-4c9d-8665-0d40cffec787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58b751e1-n95vg" Sep 11 14:55:48.175082 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:48.175055 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b63c527c-4efd-4c9d-8665-0d40cffec787-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-58b751e1-n95vg\" (UID: \"b63c527c-4efd-4c9d-8665-0d40cffec787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58b751e1-n95vg" Sep 11 14:55:48.285236 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:48.285154 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58b751e1-n95vg" Sep 11 14:55:48.559143 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:48.559115 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58b751e1-n95vg"] Sep 11 14:55:48.560463 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:55:48.560440 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb63c527c_4efd_4c9d_8665_0d40cffec787.slice/crio-4ec57da6a88160a3bf13c0395d356e5acd1d06ea47d626b2f5c3a699f16ed020 WatchSource:0}: Error finding container 4ec57da6a88160a3bf13c0395d356e5acd1d06ea47d626b2f5c3a699f16ed020: Status 404 returned error can't find the container with id 4ec57da6a88160a3bf13c0395d356e5acd1d06ea47d626b2f5c3a699f16ed020 Sep 11 14:55:49.148096 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:49.148063 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58b751e1-n95vg" event={"ID":"b63c527c-4efd-4c9d-8665-0d40cffec787","Type":"ContainerStarted","Data":"f93c5df3bd430ec9124d6b0f3953065f59eb56f59bcc6fbfd4e725cec4f8bdf1"} Sep 11 14:55:49.148427 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:49.148142 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58b751e1-n95vg" event={"ID":"b63c527c-4efd-4c9d-8665-0d40cffec787","Type":"ContainerStarted","Data":"4ec57da6a88160a3bf13c0395d356e5acd1d06ea47d626b2f5c3a699f16ed020"} Sep 11 14:55:50.156656 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:50.156623 2583 generic.go:356] "Generic (PLEG): container finished" podID="b63c527c-4efd-4c9d-8665-0d40cffec787" containerID="f93c5df3bd430ec9124d6b0f3953065f59eb56f59bcc6fbfd4e725cec4f8bdf1" exitCode=0 Sep 11 14:55:50.157080 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:50.156724 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58b751e1-n95vg" event={"ID":"b63c527c-4efd-4c9d-8665-0d40cffec787","Type":"ContainerDied","Data":"f93c5df3bd430ec9124d6b0f3953065f59eb56f59bcc6fbfd4e725cec4f8bdf1"} Sep 11 14:55:50.293144 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:50.293118 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58b751e1-n95vg" Sep 11 14:55:50.397110 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:50.397078 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b63c527c-4efd-4c9d-8665-0d40cffec787-maas-api-service-ca\") pod \"b63c527c-4efd-4c9d-8665-0d40cffec787\" (UID: \"b63c527c-4efd-4c9d-8665-0d40cffec787\") " Sep 11 14:55:50.397447 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:50.397425 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b63c527c-4efd-4c9d-8665-0d40cffec787-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b63c527c-4efd-4c9d-8665-0d40cffec787" (UID: "b63c527c-4efd-4c9d-8665-0d40cffec787"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:55:50.498894 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:50.498812 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b63c527c-4efd-4c9d-8665-0d40cffec787-maas-api-service-ca\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:55:51.164031 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:51.163993 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58b751e1-n95vg" event={"ID":"b63c527c-4efd-4c9d-8665-0d40cffec787","Type":"ContainerDied","Data":"4ec57da6a88160a3bf13c0395d356e5acd1d06ea47d626b2f5c3a699f16ed020"} Sep 11 14:55:51.164397 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:51.164047 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ec57da6a88160a3bf13c0395d356e5acd1d06ea47d626b2f5c3a699f16ed020" Sep 11 14:55:51.164397 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:55:51.164013 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58b751e1-n95vg" Sep 11 14:56:00.252871 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:00.252839 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l"] Sep 11 14:56:00.253307 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:00.253072 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l" podUID="e7e11d75-135f-483c-9ee3-c5c09c1df857" containerName="payload-processing" containerID="cri-o://c7e8dfff0454796c4391c0e80173e756dbb90e2282274d007387dcfb6f9b1245" gracePeriod=30 Sep 11 14:56:00.264378 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:00.264351 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx"] Sep 11 14:56:00.264618 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:00.264589 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx" podUID="9403bce2-551c-4523-bfe8-f3f271700b13" containerName="payload-pre-processing" containerID="cri-o://8a65c9d9e4e59ef2b861a61637a57c3107a2c724fe0101b87e8b0b8e3d9835bb" gracePeriod=30 Sep 11 14:56:00.421961 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:00.421926 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58b751e1-n95vg"] Sep 11 14:56:00.425419 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:00.425386 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-58b751e1-n95vg"] Sep 11 14:56:00.549196 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:00.549164 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l" Sep 11 14:56:00.568821 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:00.568691 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx" Sep 11 14:56:00.688278 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:00.688230 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e11d75-135f-483c-9ee3-c5c09c1df857-plugins-config-volume\") pod \"e7e11d75-135f-483c-9ee3-c5c09c1df857\" (UID: \"e7e11d75-135f-483c-9ee3-c5c09c1df857\") " Sep 11 14:56:00.688474 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:00.688314 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9rmw\" (UniqueName: \"kubernetes.io/projected/e7e11d75-135f-483c-9ee3-c5c09c1df857-kube-api-access-v9rmw\") pod \"e7e11d75-135f-483c-9ee3-c5c09c1df857\" (UID: \"e7e11d75-135f-483c-9ee3-c5c09c1df857\") " Sep 11 14:56:00.688474 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:00.688361 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dnfg\" (UniqueName: \"kubernetes.io/projected/9403bce2-551c-4523-bfe8-f3f271700b13-kube-api-access-2dnfg\") pod \"9403bce2-551c-4523-bfe8-f3f271700b13\" (UID: \"9403bce2-551c-4523-bfe8-f3f271700b13\") " Sep 11 14:56:00.688474 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:00.688443 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9403bce2-551c-4523-bfe8-f3f271700b13-plugins-config-volume\") pod \"9403bce2-551c-4523-bfe8-f3f271700b13\" (UID: \"9403bce2-551c-4523-bfe8-f3f271700b13\") " Sep 11 14:56:00.689107 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:00.689078 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7e11d75-135f-483c-9ee3-c5c09c1df857-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e7e11d75-135f-483c-9ee3-c5c09c1df857" (UID: "e7e11d75-135f-483c-9ee3-c5c09c1df857"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:56:00.689213 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:00.689098 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9403bce2-551c-4523-bfe8-f3f271700b13-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9403bce2-551c-4523-bfe8-f3f271700b13" (UID: "9403bce2-551c-4523-bfe8-f3f271700b13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:56:00.690584 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:00.690564 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e11d75-135f-483c-9ee3-c5c09c1df857-kube-api-access-v9rmw" (OuterVolumeSpecName: "kube-api-access-v9rmw") pod "e7e11d75-135f-483c-9ee3-c5c09c1df857" (UID: "e7e11d75-135f-483c-9ee3-c5c09c1df857"). InnerVolumeSpecName "kube-api-access-v9rmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:56:00.690658 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:00.690586 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9403bce2-551c-4523-bfe8-f3f271700b13-kube-api-access-2dnfg" (OuterVolumeSpecName: "kube-api-access-2dnfg") pod "9403bce2-551c-4523-bfe8-f3f271700b13" (UID: "9403bce2-551c-4523-bfe8-f3f271700b13"). InnerVolumeSpecName "kube-api-access-2dnfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:56:00.789546 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:00.789491 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e11d75-135f-483c-9ee3-c5c09c1df857-plugins-config-volume\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:56:00.789546 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:00.789513 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9rmw\" (UniqueName: \"kubernetes.io/projected/e7e11d75-135f-483c-9ee3-c5c09c1df857-kube-api-access-v9rmw\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:56:00.789546 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:00.789523 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dnfg\" (UniqueName: \"kubernetes.io/projected/9403bce2-551c-4523-bfe8-f3f271700b13-kube-api-access-2dnfg\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:56:00.789546 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:00.789531 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9403bce2-551c-4523-bfe8-f3f271700b13-plugins-config-volume\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:56:01.225980 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.225949 2583 generic.go:356] "Generic (PLEG): container finished" podID="9403bce2-551c-4523-bfe8-f3f271700b13" containerID="8a65c9d9e4e59ef2b861a61637a57c3107a2c724fe0101b87e8b0b8e3d9835bb" exitCode=0 Sep 11 14:56:01.226167 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.225994 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx" Sep 11 14:56:01.226167 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.226027 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx" event={"ID":"9403bce2-551c-4523-bfe8-f3f271700b13","Type":"ContainerDied","Data":"8a65c9d9e4e59ef2b861a61637a57c3107a2c724fe0101b87e8b0b8e3d9835bb"} Sep 11 14:56:01.226167 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.226075 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx" event={"ID":"9403bce2-551c-4523-bfe8-f3f271700b13","Type":"ContainerDied","Data":"827f171a9de3cd4798f55a209a24b1ee5a70fcfa8a1a84fa9183e9a8d823c63b"} Sep 11 14:56:01.226167 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.226096 2583 scope.go:117] "RemoveContainer" containerID="8a65c9d9e4e59ef2b861a61637a57c3107a2c724fe0101b87e8b0b8e3d9835bb" Sep 11 14:56:01.227556 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.227531 2583 generic.go:356] "Generic (PLEG): container finished" podID="e7e11d75-135f-483c-9ee3-c5c09c1df857" containerID="c7e8dfff0454796c4391c0e80173e756dbb90e2282274d007387dcfb6f9b1245" exitCode=0 Sep 11 14:56:01.227665 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.227568 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l" Sep 11 14:56:01.227665 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.227602 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l" event={"ID":"e7e11d75-135f-483c-9ee3-c5c09c1df857","Type":"ContainerDied","Data":"c7e8dfff0454796c4391c0e80173e756dbb90e2282274d007387dcfb6f9b1245"} Sep 11 14:56:01.227665 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.227645 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l" event={"ID":"e7e11d75-135f-483c-9ee3-c5c09c1df857","Type":"ContainerDied","Data":"ad47f029380522aaf52bf050d3839b956f6eff0cf5f5ce7fe169b1d223d1bd40"} Sep 11 14:56:01.236967 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.236883 2583 scope.go:117] "RemoveContainer" containerID="8a65c9d9e4e59ef2b861a61637a57c3107a2c724fe0101b87e8b0b8e3d9835bb" Sep 11 14:56:01.237286 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:56:01.237259 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a65c9d9e4e59ef2b861a61637a57c3107a2c724fe0101b87e8b0b8e3d9835bb\": container with ID starting with 8a65c9d9e4e59ef2b861a61637a57c3107a2c724fe0101b87e8b0b8e3d9835bb not found: ID does not exist" containerID="8a65c9d9e4e59ef2b861a61637a57c3107a2c724fe0101b87e8b0b8e3d9835bb" Sep 11 14:56:01.237371 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.237298 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a65c9d9e4e59ef2b861a61637a57c3107a2c724fe0101b87e8b0b8e3d9835bb"} err="failed to get container status \"8a65c9d9e4e59ef2b861a61637a57c3107a2c724fe0101b87e8b0b8e3d9835bb\": rpc error: code = NotFound desc = could not find container \"8a65c9d9e4e59ef2b861a61637a57c3107a2c724fe0101b87e8b0b8e3d9835bb\": container with ID starting with 8a65c9d9e4e59ef2b861a61637a57c3107a2c724fe0101b87e8b0b8e3d9835bb not found: ID does not exist" Sep 11 14:56:01.237371 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.237323 2583 scope.go:117] "RemoveContainer" containerID="c7e8dfff0454796c4391c0e80173e756dbb90e2282274d007387dcfb6f9b1245" Sep 11 14:56:01.246132 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.246117 2583 scope.go:117] "RemoveContainer" containerID="c7e8dfff0454796c4391c0e80173e756dbb90e2282274d007387dcfb6f9b1245" Sep 11 14:56:01.246403 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:56:01.246384 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7e8dfff0454796c4391c0e80173e756dbb90e2282274d007387dcfb6f9b1245\": container with ID starting with c7e8dfff0454796c4391c0e80173e756dbb90e2282274d007387dcfb6f9b1245 not found: ID does not exist" containerID="c7e8dfff0454796c4391c0e80173e756dbb90e2282274d007387dcfb6f9b1245" Sep 11 14:56:01.246453 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.246411 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7e8dfff0454796c4391c0e80173e756dbb90e2282274d007387dcfb6f9b1245"} err="failed to get container status \"c7e8dfff0454796c4391c0e80173e756dbb90e2282274d007387dcfb6f9b1245\": rpc error: code = NotFound desc = could not find container \"c7e8dfff0454796c4391c0e80173e756dbb90e2282274d007387dcfb6f9b1245\": container with ID starting with c7e8dfff0454796c4391c0e80173e756dbb90e2282274d007387dcfb6f9b1245 not found: ID does not exist" Sep 11 14:56:01.256826 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.256805 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l"] Sep 11 14:56:01.267454 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.267432 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-58b751e1-768ddf86f-vq98l"] Sep 11 14:56:01.284227 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.284205 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx"] Sep 11 14:56:01.287275 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.287253 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-58b751e1-6f4b5ccd-7zvfx"] Sep 11 14:56:01.331618 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.331591 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9403bce2-551c-4523-bfe8-f3f271700b13" path="/var/lib/kubelet/pods/9403bce2-551c-4523-bfe8-f3f271700b13/volumes" Sep 11 14:56:01.331965 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.331951 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b63c527c-4efd-4c9d-8665-0d40cffec787" path="/var/lib/kubelet/pods/b63c527c-4efd-4c9d-8665-0d40cffec787/volumes" Sep 11 14:56:01.332267 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.332257 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7e11d75-135f-483c-9ee3-c5c09c1df857" path="/var/lib/kubelet/pods/e7e11d75-135f-483c-9ee3-c5c09c1df857/volumes" Sep 11 14:56:01.666835 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.666804 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3219e3-8nfm5"] Sep 11 14:56:01.667201 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.667188 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b63c527c-4efd-4c9d-8665-0d40cffec787" containerName="revoke-keys" Sep 11 14:56:01.667249 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.667203 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b63c527c-4efd-4c9d-8665-0d40cffec787" containerName="revoke-keys" Sep 11 14:56:01.667249 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.667224 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7e11d75-135f-483c-9ee3-c5c09c1df857" containerName="payload-processing" Sep 11 14:56:01.667249 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.667229 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7e11d75-135f-483c-9ee3-c5c09c1df857" containerName="payload-processing" Sep 11 14:56:01.667249 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.667240 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9403bce2-551c-4523-bfe8-f3f271700b13" containerName="payload-pre-processing" Sep 11 14:56:01.667249 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.667245 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9403bce2-551c-4523-bfe8-f3f271700b13" containerName="payload-pre-processing" Sep 11 14:56:01.667401 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.667301 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9403bce2-551c-4523-bfe8-f3f271700b13" containerName="payload-pre-processing" Sep 11 14:56:01.667401 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.667309 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b63c527c-4efd-4c9d-8665-0d40cffec787" containerName="revoke-keys" Sep 11 14:56:01.667401 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.667318 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7e11d75-135f-483c-9ee3-c5c09c1df857" containerName="payload-processing" Sep 11 14:56:01.671718 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.671693 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3219e3-8nfm5" Sep 11 14:56:01.673869 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.673842 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nks8w\"" Sep 11 14:56:01.673970 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.673908 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:56:01.679330 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.679309 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3219e3-8nfm5"] Sep 11 14:56:01.799167 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.799126 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d830d299-bb6c-4e0f-92b1-4ea50c73393e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-3219e3-8nfm5\" (UID: \"d830d299-bb6c-4e0f-92b1-4ea50c73393e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3219e3-8nfm5" Sep 11 14:56:01.900181 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.900142 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d830d299-bb6c-4e0f-92b1-4ea50c73393e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-3219e3-8nfm5\" (UID: \"d830d299-bb6c-4e0f-92b1-4ea50c73393e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3219e3-8nfm5" Sep 11 14:56:01.900827 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.900808 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d830d299-bb6c-4e0f-92b1-4ea50c73393e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-3219e3-8nfm5\" (UID: \"d830d299-bb6c-4e0f-92b1-4ea50c73393e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3219e3-8nfm5" Sep 11 14:56:01.983073 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:01.982999 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3219e3-8nfm5" Sep 11 14:56:02.260368 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:02.260343 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3219e3-8nfm5"] Sep 11 14:56:02.262196 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:56:02.262170 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd830d299_bb6c_4e0f_92b1_4ea50c73393e.slice/crio-5fbfd65603a86c09e21b6ea5b7c0f458baa8d64e4325ee71990a043b6b750c16 WatchSource:0}: Error finding container 5fbfd65603a86c09e21b6ea5b7c0f458baa8d64e4325ee71990a043b6b750c16: Status 404 returned error can't find the container with id 5fbfd65603a86c09e21b6ea5b7c0f458baa8d64e4325ee71990a043b6b750c16 Sep 11 14:56:03.249420 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:03.249337 2583 generic.go:356] "Generic (PLEG): container finished" podID="d830d299-bb6c-4e0f-92b1-4ea50c73393e" containerID="984cccf4317b496f55ea5c1a4e6bc902b7ebbded4710233eb0174f5e0ea19a42" exitCode=0 Sep 11 14:56:03.249562 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:03.249415 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3219e3-8nfm5" event={"ID":"d830d299-bb6c-4e0f-92b1-4ea50c73393e","Type":"ContainerDied","Data":"984cccf4317b496f55ea5c1a4e6bc902b7ebbded4710233eb0174f5e0ea19a42"} Sep 11 14:56:03.249562 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:03.249462 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3219e3-8nfm5" event={"ID":"d830d299-bb6c-4e0f-92b1-4ea50c73393e","Type":"ContainerStarted","Data":"5fbfd65603a86c09e21b6ea5b7c0f458baa8d64e4325ee71990a043b6b750c16"} Sep 11 14:56:04.383670 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:04.383648 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3219e3-8nfm5" Sep 11 14:56:04.524008 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:04.523930 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d830d299-bb6c-4e0f-92b1-4ea50c73393e-maas-api-service-ca\") pod \"d830d299-bb6c-4e0f-92b1-4ea50c73393e\" (UID: \"d830d299-bb6c-4e0f-92b1-4ea50c73393e\") " Sep 11 14:56:04.524342 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:04.524323 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d830d299-bb6c-4e0f-92b1-4ea50c73393e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d830d299-bb6c-4e0f-92b1-4ea50c73393e" (UID: "d830d299-bb6c-4e0f-92b1-4ea50c73393e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:56:04.625444 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:04.625419 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d830d299-bb6c-4e0f-92b1-4ea50c73393e-maas-api-service-ca\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:56:05.261470 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:05.261443 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3219e3-8nfm5" Sep 11 14:56:05.261673 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:05.261476 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3219e3-8nfm5" event={"ID":"d830d299-bb6c-4e0f-92b1-4ea50c73393e","Type":"ContainerDied","Data":"5fbfd65603a86c09e21b6ea5b7c0f458baa8d64e4325ee71990a043b6b750c16"} Sep 11 14:56:05.261673 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:05.261516 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fbfd65603a86c09e21b6ea5b7c0f458baa8d64e4325ee71990a043b6b750c16" Sep 11 14:56:12.542769 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:12.542731 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq"] Sep 11 14:56:12.543250 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:12.543049 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq" podUID="c2151993-9943-413a-a5bb-c136052b15dc" containerName="payload-pre-processing" containerID="cri-o://445c5067b4b9eac525b288df0502e52fab49cea26777dbcb2b08002d28bf6fbb" gracePeriod=30 Sep 11 14:56:12.709675 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:12.709648 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3219e3-8nfm5"] Sep 11 14:56:12.714207 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:12.714183 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3219e3-8nfm5"] Sep 11 14:56:12.797114 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:12.797037 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq" Sep 11 14:56:12.899973 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:12.899943 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdj7l\" (UniqueName: \"kubernetes.io/projected/c2151993-9943-413a-a5bb-c136052b15dc-kube-api-access-sdj7l\") pod \"c2151993-9943-413a-a5bb-c136052b15dc\" (UID: \"c2151993-9943-413a-a5bb-c136052b15dc\") " Sep 11 14:56:12.900155 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:12.900128 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2151993-9943-413a-a5bb-c136052b15dc-plugins-config-volume\") pod \"c2151993-9943-413a-a5bb-c136052b15dc\" (UID: \"c2151993-9943-413a-a5bb-c136052b15dc\") " Sep 11 14:56:12.900493 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:12.900467 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2151993-9943-413a-a5bb-c136052b15dc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c2151993-9943-413a-a5bb-c136052b15dc" (UID: "c2151993-9943-413a-a5bb-c136052b15dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:56:12.902085 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:12.902061 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2151993-9943-413a-a5bb-c136052b15dc-kube-api-access-sdj7l" (OuterVolumeSpecName: "kube-api-access-sdj7l") pod "c2151993-9943-413a-a5bb-c136052b15dc" (UID: "c2151993-9943-413a-a5bb-c136052b15dc"). InnerVolumeSpecName "kube-api-access-sdj7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:56:13.001315 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:13.001286 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2151993-9943-413a-a5bb-c136052b15dc-plugins-config-volume\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:56:13.001315 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:13.001312 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdj7l\" (UniqueName: \"kubernetes.io/projected/c2151993-9943-413a-a5bb-c136052b15dc-kube-api-access-sdj7l\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:56:13.312765 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:13.312731 2583 generic.go:356] "Generic (PLEG): container finished" podID="c2151993-9943-413a-a5bb-c136052b15dc" containerID="445c5067b4b9eac525b288df0502e52fab49cea26777dbcb2b08002d28bf6fbb" exitCode=0 Sep 11 14:56:13.312765 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:13.312766 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq" Sep 11 14:56:13.312985 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:13.312812 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq" event={"ID":"c2151993-9943-413a-a5bb-c136052b15dc","Type":"ContainerDied","Data":"445c5067b4b9eac525b288df0502e52fab49cea26777dbcb2b08002d28bf6fbb"} Sep 11 14:56:13.312985 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:13.312864 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq" event={"ID":"c2151993-9943-413a-a5bb-c136052b15dc","Type":"ContainerDied","Data":"0a3b8be433d408366c0ee8082218506370c43d40d0c92dd0e7aae3cdc4d412eb"} Sep 11 14:56:13.312985 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:13.312888 2583 scope.go:117] "RemoveContainer" containerID="445c5067b4b9eac525b288df0502e52fab49cea26777dbcb2b08002d28bf6fbb" Sep 11 14:56:13.322716 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:13.322688 2583 scope.go:117] "RemoveContainer" containerID="445c5067b4b9eac525b288df0502e52fab49cea26777dbcb2b08002d28bf6fbb" Sep 11 14:56:13.322982 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:56:13.322964 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"445c5067b4b9eac525b288df0502e52fab49cea26777dbcb2b08002d28bf6fbb\": container with ID starting with 445c5067b4b9eac525b288df0502e52fab49cea26777dbcb2b08002d28bf6fbb not found: ID does not exist" containerID="445c5067b4b9eac525b288df0502e52fab49cea26777dbcb2b08002d28bf6fbb" Sep 11 14:56:13.323036 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:13.322989 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"445c5067b4b9eac525b288df0502e52fab49cea26777dbcb2b08002d28bf6fbb"} err="failed to get container status \"445c5067b4b9eac525b288df0502e52fab49cea26777dbcb2b08002d28bf6fbb\": rpc error: code = NotFound desc = could not find container \"445c5067b4b9eac525b288df0502e52fab49cea26777dbcb2b08002d28bf6fbb\": container with ID starting with 445c5067b4b9eac525b288df0502e52fab49cea26777dbcb2b08002d28bf6fbb not found: ID does not exist" Sep 11 14:56:13.330550 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:13.330522 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d830d299-bb6c-4e0f-92b1-4ea50c73393e" path="/var/lib/kubelet/pods/d830d299-bb6c-4e0f-92b1-4ea50c73393e/volumes" Sep 11 14:56:13.340925 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:13.340901 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq"] Sep 11 14:56:13.347651 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:13.347628 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3219e3-ffbbfbb86-xdblq"] Sep 11 14:56:15.331319 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:15.331288 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2151993-9943-413a-a5bb-c136052b15dc" path="/var/lib/kubelet/pods/c2151993-9943-413a-a5bb-c136052b15dc/volumes" Sep 11 14:56:41.166985 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.166895 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk"] Sep 11 14:56:41.167578 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.167554 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d830d299-bb6c-4e0f-92b1-4ea50c73393e" containerName="revoke-keys" Sep 11 14:56:41.167649 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.167582 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d830d299-bb6c-4e0f-92b1-4ea50c73393e" containerName="revoke-keys" Sep 11 14:56:41.167649 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.167624 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2151993-9943-413a-a5bb-c136052b15dc" containerName="payload-pre-processing" Sep 11 14:56:41.167649 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.167635 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2151993-9943-413a-a5bb-c136052b15dc" containerName="payload-pre-processing" Sep 11 14:56:41.167842 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.167771 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d830d299-bb6c-4e0f-92b1-4ea50c73393e" containerName="revoke-keys" Sep 11 14:56:41.167842 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.167787 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2151993-9943-413a-a5bb-c136052b15dc" containerName="payload-pre-processing" Sep 11 14:56:41.171303 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.171281 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" Sep 11 14:56:41.175272 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.175250 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-837632\"/\"premium-w4-837632-kserve-self-signed-certs\"" Sep 11 14:56:41.175397 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.175252 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-837632\"/\"kube-root-ca.crt\"" Sep 11 14:56:41.175528 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.175514 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-837632\"/\"default-dockercfg-cz84q\"" Sep 11 14:56:41.179138 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.179119 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-837632\"/\"openshift-service-ca.crt\"" Sep 11 14:56:41.184100 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.184079 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk"] Sep 11 14:56:41.271361 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.271332 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3f7c8b2-e74b-4902-a002-104b20fac586-home\") pod \"premium-w4-837632-kserve-865cf6d967-bbqnk\" (UID: \"b3f7c8b2-e74b-4902-a002-104b20fac586\") " pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" Sep 11 14:56:41.271531 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.271425 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3f7c8b2-e74b-4902-a002-104b20fac586-tls-certs\") pod \"premium-w4-837632-kserve-865cf6d967-bbqnk\" (UID: \"b3f7c8b2-e74b-4902-a002-104b20fac586\") " pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" Sep 11 14:56:41.271531 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.271456 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3f7c8b2-e74b-4902-a002-104b20fac586-model-cache\") pod \"premium-w4-837632-kserve-865cf6d967-bbqnk\" (UID: \"b3f7c8b2-e74b-4902-a002-104b20fac586\") " pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" Sep 11 14:56:41.271531 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.271470 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3f7c8b2-e74b-4902-a002-104b20fac586-tmp-dir\") pod \"premium-w4-837632-kserve-865cf6d967-bbqnk\" (UID: \"b3f7c8b2-e74b-4902-a002-104b20fac586\") " pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" Sep 11 14:56:41.271644 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.271535 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3f7c8b2-e74b-4902-a002-104b20fac586-dshm\") pod \"premium-w4-837632-kserve-865cf6d967-bbqnk\" (UID: \"b3f7c8b2-e74b-4902-a002-104b20fac586\") " pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" Sep 11 14:56:41.271644 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.271568 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wm4l\" (UniqueName: \"kubernetes.io/projected/b3f7c8b2-e74b-4902-a002-104b20fac586-kube-api-access-2wm4l\") pod \"premium-w4-837632-kserve-865cf6d967-bbqnk\" (UID: \"b3f7c8b2-e74b-4902-a002-104b20fac586\") " pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" Sep 11 14:56:41.372263 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.372228 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3f7c8b2-e74b-4902-a002-104b20fac586-model-cache\") pod \"premium-w4-837632-kserve-865cf6d967-bbqnk\" (UID: \"b3f7c8b2-e74b-4902-a002-104b20fac586\") " pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" Sep 11 14:56:41.372263 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.372267 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3f7c8b2-e74b-4902-a002-104b20fac586-tmp-dir\") pod \"premium-w4-837632-kserve-865cf6d967-bbqnk\" (UID: \"b3f7c8b2-e74b-4902-a002-104b20fac586\") " pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" Sep 11 14:56:41.372546 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.372299 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3f7c8b2-e74b-4902-a002-104b20fac586-dshm\") pod \"premium-w4-837632-kserve-865cf6d967-bbqnk\" (UID: \"b3f7c8b2-e74b-4902-a002-104b20fac586\") " pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" Sep 11 14:56:41.372546 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.372321 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wm4l\" (UniqueName: \"kubernetes.io/projected/b3f7c8b2-e74b-4902-a002-104b20fac586-kube-api-access-2wm4l\") pod \"premium-w4-837632-kserve-865cf6d967-bbqnk\" (UID: \"b3f7c8b2-e74b-4902-a002-104b20fac586\") " pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" Sep 11 14:56:41.372546 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.372359 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3f7c8b2-e74b-4902-a002-104b20fac586-home\") pod \"premium-w4-837632-kserve-865cf6d967-bbqnk\" (UID: \"b3f7c8b2-e74b-4902-a002-104b20fac586\") " pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" Sep 11 14:56:41.372546 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.372459 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3f7c8b2-e74b-4902-a002-104b20fac586-tls-certs\") pod \"premium-w4-837632-kserve-865cf6d967-bbqnk\" (UID: \"b3f7c8b2-e74b-4902-a002-104b20fac586\") " pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" Sep 11 14:56:41.372854 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.372825 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3f7c8b2-e74b-4902-a002-104b20fac586-model-cache\") pod \"premium-w4-837632-kserve-865cf6d967-bbqnk\" (UID: \"b3f7c8b2-e74b-4902-a002-104b20fac586\") " pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" Sep 11 14:56:41.373062 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.372945 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3f7c8b2-e74b-4902-a002-104b20fac586-tmp-dir\") pod \"premium-w4-837632-kserve-865cf6d967-bbqnk\" (UID: \"b3f7c8b2-e74b-4902-a002-104b20fac586\") " pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" Sep 11 14:56:41.373179 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.373128 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3f7c8b2-e74b-4902-a002-104b20fac586-home\") pod \"premium-w4-837632-kserve-865cf6d967-bbqnk\" (UID: \"b3f7c8b2-e74b-4902-a002-104b20fac586\") " pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" Sep 11 14:56:41.374819 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.374799 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3f7c8b2-e74b-4902-a002-104b20fac586-dshm\") pod \"premium-w4-837632-kserve-865cf6d967-bbqnk\" (UID: \"b3f7c8b2-e74b-4902-a002-104b20fac586\") " pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" Sep 11 14:56:41.375220 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.375193 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3f7c8b2-e74b-4902-a002-104b20fac586-tls-certs\") pod \"premium-w4-837632-kserve-865cf6d967-bbqnk\" (UID: \"b3f7c8b2-e74b-4902-a002-104b20fac586\") " pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" Sep 11 14:56:41.382757 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.382700 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wm4l\" (UniqueName: \"kubernetes.io/projected/b3f7c8b2-e74b-4902-a002-104b20fac586-kube-api-access-2wm4l\") pod \"premium-w4-837632-kserve-865cf6d967-bbqnk\" (UID: \"b3f7c8b2-e74b-4902-a002-104b20fac586\") " pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" Sep 11 14:56:41.480936 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.480837 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" Sep 11 14:56:41.759331 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:41.759304 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk"] Sep 11 14:56:41.761144 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:56:41.761118 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3f7c8b2_e74b_4902_a002_104b20fac586.slice/crio-3cace6153561e369a70d88345a4bba8d09d6403b097251695eb13c7c05cb2bd0 WatchSource:0}: Error finding container 3cace6153561e369a70d88345a4bba8d09d6403b097251695eb13c7c05cb2bd0: Status 404 returned error can't find the container with id 3cace6153561e369a70d88345a4bba8d09d6403b097251695eb13c7c05cb2bd0 Sep 11 14:56:42.482081 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:42.482043 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" event={"ID":"b3f7c8b2-e74b-4902-a002-104b20fac586","Type":"ContainerStarted","Data":"2529b55b23417c5818b4d99ba52e5e80586767bbbe67a1fe024480725de6f9d2"} Sep 11 14:56:42.482081 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:42.482084 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" event={"ID":"b3f7c8b2-e74b-4902-a002-104b20fac586","Type":"ContainerStarted","Data":"3cace6153561e369a70d88345a4bba8d09d6403b097251695eb13c7c05cb2bd0"} Sep 11 14:56:42.520054 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:42.519996 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" podStartSLOduration=1.519972763 podStartE2EDuration="1.519972763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:56:42.519374345 +0000 UTC m=+993.763360005" watchObservedRunningTime="2026-09-11 14:56:42.519972763 +0000 UTC m=+993.763958425" Sep 11 14:56:51.481990 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:51.481937 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" Sep 11 14:56:51.481990 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:51.482009 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" Sep 11 14:56:51.494755 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:51.494723 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" Sep 11 14:56:51.545408 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:56:51.545371 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" Sep 11 14:57:05.808553 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:05.808517 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf"] Sep 11 14:57:05.808971 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:05.808810 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" podUID="26130197-f0e3-4871-abb4-58a1f6331446" containerName="istio-proxy" containerID="cri-o://e7eea1c2afa8f801df4efcf8c9fcc54a55e6251bcd41bff66b93c373ee9f4439" gracePeriod=30 Sep 11 14:57:07.877052 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:07.877012 2583 patch_prober.go:28] interesting pod/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 14:57:07.877493 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:07.877082 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" podUID="26130197-f0e3-4871-abb4-58a1f6331446" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:57:11.055773 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.055749 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:57:11.138291 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.138270 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26130197-f0e3-4871-abb4-58a1f6331446-istiod-ca-cert\") pod \"26130197-f0e3-4871-abb4-58a1f6331446\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " Sep 11 14:57:11.138445 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.138305 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26130197-f0e3-4871-abb4-58a1f6331446-istio-podinfo\") pod \"26130197-f0e3-4871-abb4-58a1f6331446\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " Sep 11 14:57:11.138445 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.138331 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-istio-data\") pod \"26130197-f0e3-4871-abb4-58a1f6331446\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " Sep 11 14:57:11.138445 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.138398 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xr9ck\" (UniqueName: \"kubernetes.io/projected/26130197-f0e3-4871-abb4-58a1f6331446-kube-api-access-xr9ck\") pod \"26130197-f0e3-4871-abb4-58a1f6331446\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " Sep 11 14:57:11.138445 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.138416 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-workload-certs\") pod \"26130197-f0e3-4871-abb4-58a1f6331446\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " Sep 11 14:57:11.138675 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.138470 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-workload-socket\") pod \"26130197-f0e3-4871-abb4-58a1f6331446\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " Sep 11 14:57:11.138675 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.138507 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-istio-envoy\") pod \"26130197-f0e3-4871-abb4-58a1f6331446\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " Sep 11 14:57:11.138675 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.138551 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26130197-f0e3-4871-abb4-58a1f6331446-istio-token\") pod \"26130197-f0e3-4871-abb4-58a1f6331446\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " Sep 11 14:57:11.138675 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.138577 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-credential-socket\") pod \"26130197-f0e3-4871-abb4-58a1f6331446\" (UID: \"26130197-f0e3-4871-abb4-58a1f6331446\") " Sep 11 14:57:11.138675 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.138635 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26130197-f0e3-4871-abb4-58a1f6331446-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "26130197-f0e3-4871-abb4-58a1f6331446" (UID: "26130197-f0e3-4871-abb4-58a1f6331446"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:57:11.138974 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.138787 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "26130197-f0e3-4871-abb4-58a1f6331446" (UID: "26130197-f0e3-4871-abb4-58a1f6331446"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:57:11.138974 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.138801 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "26130197-f0e3-4871-abb4-58a1f6331446" (UID: "26130197-f0e3-4871-abb4-58a1f6331446"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:57:11.139090 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.138980 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "26130197-f0e3-4871-abb4-58a1f6331446" (UID: "26130197-f0e3-4871-abb4-58a1f6331446"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:57:11.139090 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.139004 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-workload-certs\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:57:11.139090 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.139040 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-workload-socket\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:57:11.139090 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.139046 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-istio-data" (OuterVolumeSpecName: "istio-data") pod "26130197-f0e3-4871-abb4-58a1f6331446" (UID: "26130197-f0e3-4871-abb4-58a1f6331446"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:57:11.139090 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.139057 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26130197-f0e3-4871-abb4-58a1f6331446-istiod-ca-cert\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:57:11.140597 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.140571 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26130197-f0e3-4871-abb4-58a1f6331446-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "26130197-f0e3-4871-abb4-58a1f6331446" (UID: "26130197-f0e3-4871-abb4-58a1f6331446"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:57:11.140765 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.140729 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26130197-f0e3-4871-abb4-58a1f6331446-kube-api-access-xr9ck" (OuterVolumeSpecName: "kube-api-access-xr9ck") pod "26130197-f0e3-4871-abb4-58a1f6331446" (UID: "26130197-f0e3-4871-abb4-58a1f6331446"). InnerVolumeSpecName "kube-api-access-xr9ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:57:11.141015 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.140991 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26130197-f0e3-4871-abb4-58a1f6331446-istio-token" (OuterVolumeSpecName: "istio-token") pod "26130197-f0e3-4871-abb4-58a1f6331446" (UID: "26130197-f0e3-4871-abb4-58a1f6331446"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:57:11.141015 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.140986 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "26130197-f0e3-4871-abb4-58a1f6331446" (UID: "26130197-f0e3-4871-abb4-58a1f6331446"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:57:11.240357 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.240314 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26130197-f0e3-4871-abb4-58a1f6331446-istio-podinfo\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:57:11.240357 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.240351 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-istio-data\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:57:11.240357 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.240361 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr9ck\" (UniqueName: \"kubernetes.io/projected/26130197-f0e3-4871-abb4-58a1f6331446-kube-api-access-xr9ck\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:57:11.240357 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.240372 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-istio-envoy\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:57:11.240643 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.240381 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26130197-f0e3-4871-abb4-58a1f6331446-istio-token\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:57:11.240643 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.240388 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26130197-f0e3-4871-abb4-58a1f6331446-credential-socket\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:57:11.377236 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:57:11.377192 2583 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26130197_f0e3_4871_abb4_58a1f6331446.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26130197_f0e3_4871_abb4_58a1f6331446.slice/crio-03115373058e4e9a70ea4d2f36284cb5e0ca1514b67298a03be5ddf7c81d9a7c\": RecentStats: unable to find data in memory cache]" Sep 11 14:57:11.651137 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.651102 2583 generic.go:356] "Generic (PLEG): container finished" podID="26130197-f0e3-4871-abb4-58a1f6331446" containerID="e7eea1c2afa8f801df4efcf8c9fcc54a55e6251bcd41bff66b93c373ee9f4439" exitCode=0 Sep 11 14:57:11.651318 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.651146 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" Sep 11 14:57:11.651318 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.651188 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" event={"ID":"26130197-f0e3-4871-abb4-58a1f6331446","Type":"ContainerDied","Data":"e7eea1c2afa8f801df4efcf8c9fcc54a55e6251bcd41bff66b93c373ee9f4439"} Sep 11 14:57:11.651318 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.651243 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf" event={"ID":"26130197-f0e3-4871-abb4-58a1f6331446","Type":"ContainerDied","Data":"03115373058e4e9a70ea4d2f36284cb5e0ca1514b67298a03be5ddf7c81d9a7c"} Sep 11 14:57:11.651318 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.651267 2583 scope.go:117] "RemoveContainer" containerID="e7eea1c2afa8f801df4efcf8c9fcc54a55e6251bcd41bff66b93c373ee9f4439" Sep 11 14:57:11.660639 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.660623 2583 scope.go:117] "RemoveContainer" containerID="73fb9bb192d89744f9d7008d4565d56a8274e351d767bf4d8cd2f4d1a7e67216" Sep 11 14:57:11.670256 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.670237 2583 scope.go:117] "RemoveContainer" containerID="e7eea1c2afa8f801df4efcf8c9fcc54a55e6251bcd41bff66b93c373ee9f4439" Sep 11 14:57:11.670592 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:57:11.670563 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7eea1c2afa8f801df4efcf8c9fcc54a55e6251bcd41bff66b93c373ee9f4439\": container with ID starting with e7eea1c2afa8f801df4efcf8c9fcc54a55e6251bcd41bff66b93c373ee9f4439 not found: ID does not exist" containerID="e7eea1c2afa8f801df4efcf8c9fcc54a55e6251bcd41bff66b93c373ee9f4439" Sep 11 14:57:11.670734 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.670596 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7eea1c2afa8f801df4efcf8c9fcc54a55e6251bcd41bff66b93c373ee9f4439"} err="failed to get container status \"e7eea1c2afa8f801df4efcf8c9fcc54a55e6251bcd41bff66b93c373ee9f4439\": rpc error: code = NotFound desc = could not find container \"e7eea1c2afa8f801df4efcf8c9fcc54a55e6251bcd41bff66b93c373ee9f4439\": container with ID starting with e7eea1c2afa8f801df4efcf8c9fcc54a55e6251bcd41bff66b93c373ee9f4439 not found: ID does not exist" Sep 11 14:57:11.670734 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.670618 2583 scope.go:117] "RemoveContainer" containerID="73fb9bb192d89744f9d7008d4565d56a8274e351d767bf4d8cd2f4d1a7e67216" Sep 11 14:57:11.671025 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:57:11.670933 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73fb9bb192d89744f9d7008d4565d56a8274e351d767bf4d8cd2f4d1a7e67216\": container with ID starting with 73fb9bb192d89744f9d7008d4565d56a8274e351d767bf4d8cd2f4d1a7e67216 not found: ID does not exist" containerID="73fb9bb192d89744f9d7008d4565d56a8274e351d767bf4d8cd2f4d1a7e67216" Sep 11 14:57:11.671025 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.670962 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73fb9bb192d89744f9d7008d4565d56a8274e351d767bf4d8cd2f4d1a7e67216"} err="failed to get container status \"73fb9bb192d89744f9d7008d4565d56a8274e351d767bf4d8cd2f4d1a7e67216\": rpc error: code = NotFound desc = could not find container \"73fb9bb192d89744f9d7008d4565d56a8274e351d767bf4d8cd2f4d1a7e67216\": container with ID starting with 73fb9bb192d89744f9d7008d4565d56a8274e351d767bf4d8cd2f4d1a7e67216 not found: ID does not exist" Sep 11 14:57:11.671025 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.670992 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf"] Sep 11 14:57:11.673635 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:11.673617 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-3219e3-openshift-default-587759d66d-9zqzf"] Sep 11 14:57:13.331033 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:13.331001 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26130197-f0e3-4871-abb4-58a1f6331446" path="/var/lib/kubelet/pods/26130197-f0e3-4871-abb4-58a1f6331446/volumes" Sep 11 14:57:14.535313 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.535282 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx"] Sep 11 14:57:14.535669 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.535653 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26130197-f0e3-4871-abb4-58a1f6331446" containerName="istio-proxy" Sep 11 14:57:14.535669 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.535663 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="26130197-f0e3-4871-abb4-58a1f6331446" containerName="istio-proxy" Sep 11 14:57:14.535775 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.535672 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26130197-f0e3-4871-abb4-58a1f6331446" containerName="istio-proxy" Sep 11 14:57:14.535775 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.535678 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="26130197-f0e3-4871-abb4-58a1f6331446" containerName="istio-proxy" Sep 11 14:57:14.535775 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.535764 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="26130197-f0e3-4871-abb4-58a1f6331446" containerName="istio-proxy" Sep 11 14:57:14.535775 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.535773 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="26130197-f0e3-4871-abb4-58a1f6331446" containerName="istio-proxy" Sep 11 14:57:14.540408 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.540391 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx" Sep 11 14:57:14.542976 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.542952 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-87d086\"" Sep 11 14:57:14.543061 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.542995 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-87d086-dockercfg-pl6lr\"" Sep 11 14:57:14.552102 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.552079 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx"] Sep 11 14:57:14.558769 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.558748 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl"] Sep 11 14:57:14.562114 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.562093 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl" Sep 11 14:57:14.575862 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.575839 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl"] Sep 11 14:57:14.677080 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.677051 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5005ad13-1c15-4f0a-8787-48b604de27f2-plugins-config-volume\") pod \"payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl\" (UID: \"5005ad13-1c15-4f0a-8787-48b604de27f2\") " pod="openshift-ingress/payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl" Sep 11 14:57:14.677211 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.677101 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fab6801d-4d6a-4fb0-81b7-822f7192051e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx\" (UID: \"fab6801d-4d6a-4fb0-81b7-822f7192051e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx" Sep 11 14:57:14.677211 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.677131 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d9q4\" (UniqueName: \"kubernetes.io/projected/5005ad13-1c15-4f0a-8787-48b604de27f2-kube-api-access-4d9q4\") pod \"payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl\" (UID: \"5005ad13-1c15-4f0a-8787-48b604de27f2\") " pod="openshift-ingress/payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl" Sep 11 14:57:14.677211 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.677160 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rs7k\" (UniqueName: \"kubernetes.io/projected/fab6801d-4d6a-4fb0-81b7-822f7192051e-kube-api-access-6rs7k\") pod \"payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx\" (UID: \"fab6801d-4d6a-4fb0-81b7-822f7192051e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx" Sep 11 14:57:14.778333 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.778301 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5005ad13-1c15-4f0a-8787-48b604de27f2-plugins-config-volume\") pod \"payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl\" (UID: \"5005ad13-1c15-4f0a-8787-48b604de27f2\") " pod="openshift-ingress/payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl" Sep 11 14:57:14.778469 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.778355 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fab6801d-4d6a-4fb0-81b7-822f7192051e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx\" (UID: \"fab6801d-4d6a-4fb0-81b7-822f7192051e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx" Sep 11 14:57:14.778559 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.778529 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d9q4\" (UniqueName: \"kubernetes.io/projected/5005ad13-1c15-4f0a-8787-48b604de27f2-kube-api-access-4d9q4\") pod \"payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl\" (UID: \"5005ad13-1c15-4f0a-8787-48b604de27f2\") " pod="openshift-ingress/payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl" Sep 11 14:57:14.778635 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.778620 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rs7k\" (UniqueName: \"kubernetes.io/projected/fab6801d-4d6a-4fb0-81b7-822f7192051e-kube-api-access-6rs7k\") pod \"payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx\" (UID: \"fab6801d-4d6a-4fb0-81b7-822f7192051e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx" Sep 11 14:57:14.779075 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.779059 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5005ad13-1c15-4f0a-8787-48b604de27f2-plugins-config-volume\") pod \"payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl\" (UID: \"5005ad13-1c15-4f0a-8787-48b604de27f2\") " pod="openshift-ingress/payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl" Sep 11 14:57:14.779128 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.779085 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fab6801d-4d6a-4fb0-81b7-822f7192051e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx\" (UID: \"fab6801d-4d6a-4fb0-81b7-822f7192051e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx" Sep 11 14:57:14.788112 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.788055 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rs7k\" (UniqueName: \"kubernetes.io/projected/fab6801d-4d6a-4fb0-81b7-822f7192051e-kube-api-access-6rs7k\") pod \"payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx\" (UID: \"fab6801d-4d6a-4fb0-81b7-822f7192051e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx" Sep 11 14:57:14.788211 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.788179 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d9q4\" (UniqueName: \"kubernetes.io/projected/5005ad13-1c15-4f0a-8787-48b604de27f2-kube-api-access-4d9q4\") pod \"payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl\" (UID: \"5005ad13-1c15-4f0a-8787-48b604de27f2\") " pod="openshift-ingress/payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl" Sep 11 14:57:14.851939 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.851899 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx" Sep 11 14:57:14.872277 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:14.872242 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl" Sep 11 14:57:15.132127 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:15.132101 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx"] Sep 11 14:57:15.133439 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:57:15.133414 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab6801d_4d6a_4fb0_81b7_822f7192051e.slice/crio-0b907c919cdf5a89facec342e29325a99a3adefa4845fb2296d01696ff629890 WatchSource:0}: Error finding container 0b907c919cdf5a89facec342e29325a99a3adefa4845fb2296d01696ff629890: Status 404 returned error can't find the container with id 0b907c919cdf5a89facec342e29325a99a3adefa4845fb2296d01696ff629890 Sep 11 14:57:15.162578 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:15.162552 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl"] Sep 11 14:57:15.164639 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:57:15.164614 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5005ad13_1c15_4f0a_8787_48b604de27f2.slice/crio-0bbd441a118e52cc122a18f8fbab247c64faf188e5f0073cd5ef85be1ca40e28 WatchSource:0}: Error finding container 0bbd441a118e52cc122a18f8fbab247c64faf188e5f0073cd5ef85be1ca40e28: Status 404 returned error can't find the container with id 0bbd441a118e52cc122a18f8fbab247c64faf188e5f0073cd5ef85be1ca40e28 Sep 11 14:57:15.680775 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:15.680739 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl" event={"ID":"5005ad13-1c15-4f0a-8787-48b604de27f2","Type":"ContainerStarted","Data":"b347c665705ca1973169f72b231f61fe9bf0efd4c0d9027202ae04eae17c2e5c"} Sep 11 14:57:15.681248 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:15.680804 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl" event={"ID":"5005ad13-1c15-4f0a-8787-48b604de27f2","Type":"ContainerStarted","Data":"0bbd441a118e52cc122a18f8fbab247c64faf188e5f0073cd5ef85be1ca40e28"} Sep 11 14:57:15.681248 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:15.680947 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl" Sep 11 14:57:15.682415 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:15.682393 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx" event={"ID":"fab6801d-4d6a-4fb0-81b7-822f7192051e","Type":"ContainerStarted","Data":"93c42ac95a2259ce4e798dcaff37638add54511fb0f7267b0a92cd260b59826b"} Sep 11 14:57:15.682524 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:15.682434 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx" event={"ID":"fab6801d-4d6a-4fb0-81b7-822f7192051e","Type":"ContainerStarted","Data":"0b907c919cdf5a89facec342e29325a99a3adefa4845fb2296d01696ff629890"} Sep 11 14:57:15.682524 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:15.682467 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx" Sep 11 14:57:15.699922 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:15.699878 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl" podStartSLOduration=1.699865376 podStartE2EDuration="1.699865376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:57:15.698485713 +0000 UTC m=+1026.942471374" watchObservedRunningTime="2026-09-11 14:57:15.699865376 +0000 UTC m=+1026.943851066" Sep 11 14:57:15.721422 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:15.721379 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx" podStartSLOduration=1.7213611690000001 podStartE2EDuration="1.721361169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:57:15.718233565 +0000 UTC m=+1026.962219221" watchObservedRunningTime="2026-09-11 14:57:15.721361169 +0000 UTC m=+1026.965346829" Sep 11 14:57:26.690868 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:26.690840 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx" Sep 11 14:57:26.691308 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:26.690906 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl" Sep 11 14:57:54.546456 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:54.546414 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94"] Sep 11 14:57:54.550420 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:54.550401 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94" Sep 11 14:57:54.553161 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:54.553140 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-577852b1\"" Sep 11 14:57:54.558646 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:54.558624 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc"] Sep 11 14:57:54.561979 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:54.561963 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc" Sep 11 14:57:54.564500 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:54.564482 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-577852b1-dockercfg-b457m\"" Sep 11 14:57:54.566778 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:54.566759 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94"] Sep 11 14:57:54.585028 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:54.585000 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc"] Sep 11 14:57:54.654442 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:54.654389 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ef38a9-fa6c-4d00-9793-cd877453a927-plugins-config-volume\") pod \"payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc\" (UID: \"c2ef38a9-fa6c-4d00-9793-cd877453a927\") " pod="openshift-ingress/payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc" Sep 11 14:57:54.654637 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:54.654478 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjvqj\" (UniqueName: \"kubernetes.io/projected/c2ef38a9-fa6c-4d00-9793-cd877453a927-kube-api-access-mjvqj\") pod \"payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc\" (UID: \"c2ef38a9-fa6c-4d00-9793-cd877453a927\") " pod="openshift-ingress/payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc" Sep 11 14:57:54.654637 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:54.654515 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a4d210-abcb-4372-8c3c-8610069ebe44-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94\" (UID: \"30a4d210-abcb-4372-8c3c-8610069ebe44\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94" Sep 11 14:57:54.654637 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:54.654552 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sbmj\" (UniqueName: \"kubernetes.io/projected/30a4d210-abcb-4372-8c3c-8610069ebe44-kube-api-access-6sbmj\") pod \"payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94\" (UID: \"30a4d210-abcb-4372-8c3c-8610069ebe44\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94" Sep 11 14:57:54.755416 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:54.755381 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ef38a9-fa6c-4d00-9793-cd877453a927-plugins-config-volume\") pod \"payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc\" (UID: \"c2ef38a9-fa6c-4d00-9793-cd877453a927\") " pod="openshift-ingress/payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc" Sep 11 14:57:54.755582 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:54.755434 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjvqj\" (UniqueName: \"kubernetes.io/projected/c2ef38a9-fa6c-4d00-9793-cd877453a927-kube-api-access-mjvqj\") pod \"payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc\" (UID: \"c2ef38a9-fa6c-4d00-9793-cd877453a927\") " pod="openshift-ingress/payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc" Sep 11 14:57:54.755582 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:54.755454 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a4d210-abcb-4372-8c3c-8610069ebe44-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94\" (UID: \"30a4d210-abcb-4372-8c3c-8610069ebe44\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94" Sep 11 14:57:54.755582 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:54.755470 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sbmj\" (UniqueName: \"kubernetes.io/projected/30a4d210-abcb-4372-8c3c-8610069ebe44-kube-api-access-6sbmj\") pod \"payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94\" (UID: \"30a4d210-abcb-4372-8c3c-8610069ebe44\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94" Sep 11 14:57:54.756165 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:54.756143 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ef38a9-fa6c-4d00-9793-cd877453a927-plugins-config-volume\") pod \"payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc\" (UID: \"c2ef38a9-fa6c-4d00-9793-cd877453a927\") " pod="openshift-ingress/payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc" Sep 11 14:57:54.756205 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:54.756177 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a4d210-abcb-4372-8c3c-8610069ebe44-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94\" (UID: \"30a4d210-abcb-4372-8c3c-8610069ebe44\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94" Sep 11 14:57:54.765237 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:54.765214 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjvqj\" (UniqueName: \"kubernetes.io/projected/c2ef38a9-fa6c-4d00-9793-cd877453a927-kube-api-access-mjvqj\") pod \"payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc\" (UID: \"c2ef38a9-fa6c-4d00-9793-cd877453a927\") " pod="openshift-ingress/payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc" Sep 11 14:57:54.765353 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:54.765314 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sbmj\" (UniqueName: \"kubernetes.io/projected/30a4d210-abcb-4372-8c3c-8610069ebe44-kube-api-access-6sbmj\") pod \"payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94\" (UID: \"30a4d210-abcb-4372-8c3c-8610069ebe44\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94" Sep 11 14:57:54.861597 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:54.861524 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94" Sep 11 14:57:54.872650 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:54.872628 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc" Sep 11 14:57:55.148213 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:55.148185 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94"] Sep 11 14:57:55.149493 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:57:55.149461 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a4d210_abcb_4372_8c3c_8610069ebe44.slice/crio-290e47adabc1d896b20eb2c1464a53b6bb2a075ccf0d35b1fcc25f0cca93f1d6 WatchSource:0}: Error finding container 290e47adabc1d896b20eb2c1464a53b6bb2a075ccf0d35b1fcc25f0cca93f1d6: Status 404 returned error can't find the container with id 290e47adabc1d896b20eb2c1464a53b6bb2a075ccf0d35b1fcc25f0cca93f1d6 Sep 11 14:57:55.164317 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:57:55.164290 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ef38a9_fa6c_4d00_9793_cd877453a927.slice/crio-2e4d6947f18d184fd72438afa630ac7c0c498a1c5f4cadd9a6cd1644402c73ce WatchSource:0}: Error finding container 2e4d6947f18d184fd72438afa630ac7c0c498a1c5f4cadd9a6cd1644402c73ce: Status 404 returned error can't find the container with id 2e4d6947f18d184fd72438afa630ac7c0c498a1c5f4cadd9a6cd1644402c73ce Sep 11 14:57:55.164435 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:55.164356 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc"] Sep 11 14:57:55.926158 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:55.926115 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc" event={"ID":"c2ef38a9-fa6c-4d00-9793-cd877453a927","Type":"ContainerStarted","Data":"b10995e185b398211fed021fe852d4a1d7c64981620999aea5fc4b5792660016"} Sep 11 14:57:55.926606 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:55.926475 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc" event={"ID":"c2ef38a9-fa6c-4d00-9793-cd877453a927","Type":"ContainerStarted","Data":"2e4d6947f18d184fd72438afa630ac7c0c498a1c5f4cadd9a6cd1644402c73ce"} Sep 11 14:57:55.926737 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:55.926640 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc" Sep 11 14:57:55.940744 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:55.940696 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94" event={"ID":"30a4d210-abcb-4372-8c3c-8610069ebe44","Type":"ContainerStarted","Data":"19a663585afe3f570f7c0c59c5b800ab4f095b35fd7020781b6ef14ceb52bd73"} Sep 11 14:57:55.940902 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:55.940793 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94" event={"ID":"30a4d210-abcb-4372-8c3c-8610069ebe44","Type":"ContainerStarted","Data":"290e47adabc1d896b20eb2c1464a53b6bb2a075ccf0d35b1fcc25f0cca93f1d6"} Sep 11 14:57:55.941227 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:55.941210 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94" Sep 11 14:57:55.949188 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:55.949123 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc" podStartSLOduration=1.949103119 podStartE2EDuration="1.949103119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:57:55.946729271 +0000 UTC m=+1067.190714926" watchObservedRunningTime="2026-09-11 14:57:55.949103119 +0000 UTC m=+1067.193088780" Sep 11 14:57:55.965637 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:57:55.965580 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94" podStartSLOduration=1.965540273 podStartE2EDuration="1.965540273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:57:55.962996003 +0000 UTC m=+1067.206981663" watchObservedRunningTime="2026-09-11 14:57:55.965540273 +0000 UTC m=+1067.209525935" Sep 11 14:58:01.102316 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:01.102273 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns"] Sep 11 14:58:01.106286 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:01.106262 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" Sep 11 14:58:01.108691 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:01.108670 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-837632\"/\"e2e-conflict-a-372348b4-kserve-self-signed-certs\"" Sep 11 14:58:01.116956 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:01.116929 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns"] Sep 11 14:58:01.214197 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:01.214162 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-model-cache\") pod \"e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns\" (UID: \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" Sep 11 14:58:01.214197 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:01.214200 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-home\") pod \"e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns\" (UID: \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" Sep 11 14:58:01.214395 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:01.214235 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bggz7\" (UniqueName: \"kubernetes.io/projected/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-kube-api-access-bggz7\") pod \"e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns\" (UID: \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" Sep 11 14:58:01.214395 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:01.214334 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-dshm\") pod \"e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns\" (UID: \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" Sep 11 14:58:01.214395 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:01.214378 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-tmp-dir\") pod \"e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns\" (UID: \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" Sep 11 14:58:01.214494 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:01.214429 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-tls-certs\") pod \"e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns\" (UID: \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" Sep 11 14:58:01.315979 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:01.315938 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-model-cache\") pod \"e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns\" (UID: \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" Sep 11 14:58:01.315979 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:01.315987 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-home\") pod \"e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns\" (UID: \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" Sep 11 14:58:01.316215 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:01.316028 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bggz7\" (UniqueName: \"kubernetes.io/projected/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-kube-api-access-bggz7\") pod \"e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns\" (UID: \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" Sep 11 14:58:01.316215 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:01.316079 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-dshm\") pod \"e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns\" (UID: \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" Sep 11 14:58:01.316215 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:01.316127 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-tmp-dir\") pod \"e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns\" (UID: \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" Sep 11 14:58:01.316215 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:01.316157 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-tls-certs\") pod \"e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns\" (UID: \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" Sep 11 14:58:01.316434 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:01.316349 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-model-cache\") pod \"e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns\" (UID: \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" Sep 11 14:58:01.316434 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:01.316410 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-home\") pod \"e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns\" (UID: \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" Sep 11 14:58:01.316542 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:01.316445 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-tmp-dir\") pod \"e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns\" (UID: \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" Sep 11 14:58:01.318418 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:01.318396 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-dshm\") pod \"e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns\" (UID: \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" Sep 11 14:58:01.318679 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:01.318663 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-tls-certs\") pod \"e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns\" (UID: \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" Sep 11 14:58:01.326545 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:01.326513 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bggz7\" (UniqueName: \"kubernetes.io/projected/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-kube-api-access-bggz7\") pod \"e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns\" (UID: \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" Sep 11 14:58:01.416883 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:01.416852 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" Sep 11 14:58:01.705194 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:01.705170 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns"] Sep 11 14:58:01.706644 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:58:01.706615 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44b1ed25_fcc4_4eea_9d4b_725e66db1b61.slice/crio-61bcf12606d6e5b3590aa327c9d11f9e9885af1150ad5e16c2a76500b48659bb WatchSource:0}: Error finding container 61bcf12606d6e5b3590aa327c9d11f9e9885af1150ad5e16c2a76500b48659bb: Status 404 returned error can't find the container with id 61bcf12606d6e5b3590aa327c9d11f9e9885af1150ad5e16c2a76500b48659bb Sep 11 14:58:01.983816 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:01.983696 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" event={"ID":"44b1ed25-fcc4-4eea-9d4b-725e66db1b61","Type":"ContainerStarted","Data":"da2d844be0aa2962b037ca05c25e5097b5d2ca180a061a2b8102e8de84f7ea41"} Sep 11 14:58:01.983816 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:01.983762 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" event={"ID":"44b1ed25-fcc4-4eea-9d4b-725e66db1b61","Type":"ContainerStarted","Data":"61bcf12606d6e5b3590aa327c9d11f9e9885af1150ad5e16c2a76500b48659bb"} Sep 11 14:58:02.003975 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:02.003928 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" podStartSLOduration=1.003914171 podStartE2EDuration="1.003914171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:58:02.001258761 +0000 UTC m=+1073.245244422" watchObservedRunningTime="2026-09-11 14:58:02.003914171 +0000 UTC m=+1073.247899886" Sep 11 14:58:06.948892 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:06.948858 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc" Sep 11 14:58:07.955114 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:07.955081 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94" Sep 11 14:58:11.417347 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:11.417304 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" Sep 11 14:58:11.417967 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:11.417748 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" Sep 11 14:58:11.431644 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:11.431621 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" Sep 11 14:58:12.056422 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:12.056397 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" Sep 11 14:58:27.646352 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:27.646323 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-577852b1-l7lh6"] Sep 11 14:58:27.650990 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:27.650971 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-577852b1-l7lh6" Sep 11 14:58:27.653517 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:27.653493 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:58:27.653637 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:27.653545 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nks8w\"" Sep 11 14:58:27.659196 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:27.659173 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-577852b1-l7lh6"] Sep 11 14:58:27.762191 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:27.762159 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb076503-cce1-429c-b2b0-d5d581ea4c10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-577852b1-l7lh6\" (UID: \"bb076503-cce1-429c-b2b0-d5d581ea4c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-577852b1-l7lh6" Sep 11 14:58:27.863601 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:27.863568 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb076503-cce1-429c-b2b0-d5d581ea4c10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-577852b1-l7lh6\" (UID: \"bb076503-cce1-429c-b2b0-d5d581ea4c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-577852b1-l7lh6" Sep 11 14:58:27.864157 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:27.864140 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb076503-cce1-429c-b2b0-d5d581ea4c10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-577852b1-l7lh6\" (UID: \"bb076503-cce1-429c-b2b0-d5d581ea4c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-577852b1-l7lh6" Sep 11 14:58:27.962986 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:27.962923 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-577852b1-l7lh6" Sep 11 14:58:28.238984 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:28.238956 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-577852b1-l7lh6"] Sep 11 14:58:28.240915 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:58:28.240887 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb076503_cce1_429c_b2b0_d5d581ea4c10.slice/crio-3b9ddc8f6b094ddc18fd9c0e1b925d8193a63211bd0d4f5b4cfaa1f23a666893 WatchSource:0}: Error finding container 3b9ddc8f6b094ddc18fd9c0e1b925d8193a63211bd0d4f5b4cfaa1f23a666893: Status 404 returned error can't find the container with id 3b9ddc8f6b094ddc18fd9c0e1b925d8193a63211bd0d4f5b4cfaa1f23a666893 Sep 11 14:58:29.146143 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:29.146114 2583 generic.go:356] "Generic (PLEG): container finished" podID="bb076503-cce1-429c-b2b0-d5d581ea4c10" containerID="dacf3650f8837589615aeacf65c6036339c7c067392343575ff5391b5781765e" exitCode=0 Sep 11 14:58:29.146505 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:29.146174 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-577852b1-l7lh6" event={"ID":"bb076503-cce1-429c-b2b0-d5d581ea4c10","Type":"ContainerDied","Data":"dacf3650f8837589615aeacf65c6036339c7c067392343575ff5391b5781765e"} Sep 11 14:58:29.146505 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:29.146209 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-577852b1-l7lh6" event={"ID":"bb076503-cce1-429c-b2b0-d5d581ea4c10","Type":"ContainerStarted","Data":"3b9ddc8f6b094ddc18fd9c0e1b925d8193a63211bd0d4f5b4cfaa1f23a666893"} Sep 11 14:58:30.295764 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:30.295740 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-577852b1-l7lh6" Sep 11 14:58:30.387818 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:30.387777 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb076503-cce1-429c-b2b0-d5d581ea4c10-maas-api-service-ca\") pod \"bb076503-cce1-429c-b2b0-d5d581ea4c10\" (UID: \"bb076503-cce1-429c-b2b0-d5d581ea4c10\") " Sep 11 14:58:30.388121 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:30.388097 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb076503-cce1-429c-b2b0-d5d581ea4c10-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bb076503-cce1-429c-b2b0-d5d581ea4c10" (UID: "bb076503-cce1-429c-b2b0-d5d581ea4c10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:58:30.388402 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:30.388322 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb076503-cce1-429c-b2b0-d5d581ea4c10-maas-api-service-ca\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:58:31.163012 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:31.162981 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-577852b1-l7lh6" Sep 11 14:58:31.163181 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:31.163006 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-577852b1-l7lh6" event={"ID":"bb076503-cce1-429c-b2b0-d5d581ea4c10","Type":"ContainerDied","Data":"3b9ddc8f6b094ddc18fd9c0e1b925d8193a63211bd0d4f5b4cfaa1f23a666893"} Sep 11 14:58:31.163181 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:31.163055 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b9ddc8f6b094ddc18fd9c0e1b925d8193a63211bd0d4f5b4cfaa1f23a666893" Sep 11 14:58:38.252594 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:38.252550 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc"] Sep 11 14:58:38.253085 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:38.252825 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc" podUID="c2ef38a9-fa6c-4d00-9793-cd877453a927" containerName="payload-processing" containerID="cri-o://b10995e185b398211fed021fe852d4a1d7c64981620999aea5fc4b5792660016" gracePeriod=30 Sep 11 14:58:38.259832 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:38.259808 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94"] Sep 11 14:58:38.260157 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:38.260026 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94" podUID="30a4d210-abcb-4372-8c3c-8610069ebe44" containerName="payload-pre-processing" containerID="cri-o://19a663585afe3f570f7c0c59c5b800ab4f095b35fd7020781b6ef14ceb52bd73" gracePeriod=30 Sep 11 14:58:38.523543 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:38.523146 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-577852b1-l7lh6"] Sep 11 14:58:38.526342 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:38.526316 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-577852b1-l7lh6"] Sep 11 14:58:38.528173 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:38.528149 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc" Sep 11 14:58:38.558546 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:38.558523 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94" Sep 11 14:58:38.666962 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:38.666928 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sbmj\" (UniqueName: \"kubernetes.io/projected/30a4d210-abcb-4372-8c3c-8610069ebe44-kube-api-access-6sbmj\") pod \"30a4d210-abcb-4372-8c3c-8610069ebe44\" (UID: \"30a4d210-abcb-4372-8c3c-8610069ebe44\") " Sep 11 14:58:38.667153 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:38.666972 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a4d210-abcb-4372-8c3c-8610069ebe44-plugins-config-volume\") pod \"30a4d210-abcb-4372-8c3c-8610069ebe44\" (UID: \"30a4d210-abcb-4372-8c3c-8610069ebe44\") " Sep 11 14:58:38.667153 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:38.666996 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ef38a9-fa6c-4d00-9793-cd877453a927-plugins-config-volume\") pod \"c2ef38a9-fa6c-4d00-9793-cd877453a927\" (UID: \"c2ef38a9-fa6c-4d00-9793-cd877453a927\") " Sep 11 14:58:38.667153 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:38.667079 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjvqj\" (UniqueName: \"kubernetes.io/projected/c2ef38a9-fa6c-4d00-9793-cd877453a927-kube-api-access-mjvqj\") pod \"c2ef38a9-fa6c-4d00-9793-cd877453a927\" (UID: \"c2ef38a9-fa6c-4d00-9793-cd877453a927\") " Sep 11 14:58:38.667398 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:38.667368 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30a4d210-abcb-4372-8c3c-8610069ebe44-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "30a4d210-abcb-4372-8c3c-8610069ebe44" (UID: "30a4d210-abcb-4372-8c3c-8610069ebe44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:58:38.667511 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:38.667448 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2ef38a9-fa6c-4d00-9793-cd877453a927-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c2ef38a9-fa6c-4d00-9793-cd877453a927" (UID: "c2ef38a9-fa6c-4d00-9793-cd877453a927"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:58:38.669121 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:38.669102 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30a4d210-abcb-4372-8c3c-8610069ebe44-kube-api-access-6sbmj" (OuterVolumeSpecName: "kube-api-access-6sbmj") pod "30a4d210-abcb-4372-8c3c-8610069ebe44" (UID: "30a4d210-abcb-4372-8c3c-8610069ebe44"). InnerVolumeSpecName "kube-api-access-6sbmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:58:38.669190 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:38.669158 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ef38a9-fa6c-4d00-9793-cd877453a927-kube-api-access-mjvqj" (OuterVolumeSpecName: "kube-api-access-mjvqj") pod "c2ef38a9-fa6c-4d00-9793-cd877453a927" (UID: "c2ef38a9-fa6c-4d00-9793-cd877453a927"). InnerVolumeSpecName "kube-api-access-mjvqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:58:38.768923 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:38.768848 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sbmj\" (UniqueName: \"kubernetes.io/projected/30a4d210-abcb-4372-8c3c-8610069ebe44-kube-api-access-6sbmj\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:58:38.768923 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:38.768873 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a4d210-abcb-4372-8c3c-8610069ebe44-plugins-config-volume\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:58:38.768923 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:38.768883 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ef38a9-fa6c-4d00-9793-cd877453a927-plugins-config-volume\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:58:38.768923 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:38.768892 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjvqj\" (UniqueName: \"kubernetes.io/projected/c2ef38a9-fa6c-4d00-9793-cd877453a927-kube-api-access-mjvqj\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:58:39.214505 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:39.214467 2583 generic.go:356] "Generic (PLEG): container finished" podID="30a4d210-abcb-4372-8c3c-8610069ebe44" containerID="19a663585afe3f570f7c0c59c5b800ab4f095b35fd7020781b6ef14ceb52bd73" exitCode=0 Sep 11 14:58:39.214693 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:39.214513 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94" Sep 11 14:58:39.214693 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:39.214562 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94" event={"ID":"30a4d210-abcb-4372-8c3c-8610069ebe44","Type":"ContainerDied","Data":"19a663585afe3f570f7c0c59c5b800ab4f095b35fd7020781b6ef14ceb52bd73"} Sep 11 14:58:39.214693 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:39.214624 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94" event={"ID":"30a4d210-abcb-4372-8c3c-8610069ebe44","Type":"ContainerDied","Data":"290e47adabc1d896b20eb2c1464a53b6bb2a075ccf0d35b1fcc25f0cca93f1d6"} Sep 11 14:58:39.214693 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:39.214654 2583 scope.go:117] "RemoveContainer" containerID="19a663585afe3f570f7c0c59c5b800ab4f095b35fd7020781b6ef14ceb52bd73" Sep 11 14:58:39.216067 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:39.216037 2583 generic.go:356] "Generic (PLEG): container finished" podID="c2ef38a9-fa6c-4d00-9793-cd877453a927" containerID="b10995e185b398211fed021fe852d4a1d7c64981620999aea5fc4b5792660016" exitCode=0 Sep 11 14:58:39.216162 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:39.216076 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc" Sep 11 14:58:39.216162 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:39.216080 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc" event={"ID":"c2ef38a9-fa6c-4d00-9793-cd877453a927","Type":"ContainerDied","Data":"b10995e185b398211fed021fe852d4a1d7c64981620999aea5fc4b5792660016"} Sep 11 14:58:39.216162 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:39.216142 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc" event={"ID":"c2ef38a9-fa6c-4d00-9793-cd877453a927","Type":"ContainerDied","Data":"2e4d6947f18d184fd72438afa630ac7c0c498a1c5f4cadd9a6cd1644402c73ce"} Sep 11 14:58:39.226774 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:39.226755 2583 scope.go:117] "RemoveContainer" containerID="19a663585afe3f570f7c0c59c5b800ab4f095b35fd7020781b6ef14ceb52bd73" Sep 11 14:58:39.227044 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:58:39.227024 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19a663585afe3f570f7c0c59c5b800ab4f095b35fd7020781b6ef14ceb52bd73\": container with ID starting with 19a663585afe3f570f7c0c59c5b800ab4f095b35fd7020781b6ef14ceb52bd73 not found: ID does not exist" containerID="19a663585afe3f570f7c0c59c5b800ab4f095b35fd7020781b6ef14ceb52bd73" Sep 11 14:58:39.227136 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:39.227050 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19a663585afe3f570f7c0c59c5b800ab4f095b35fd7020781b6ef14ceb52bd73"} err="failed to get container status \"19a663585afe3f570f7c0c59c5b800ab4f095b35fd7020781b6ef14ceb52bd73\": rpc error: code = NotFound desc = could not find container \"19a663585afe3f570f7c0c59c5b800ab4f095b35fd7020781b6ef14ceb52bd73\": container with ID starting with 19a663585afe3f570f7c0c59c5b800ab4f095b35fd7020781b6ef14ceb52bd73 not found: ID does not exist" Sep 11 14:58:39.227136 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:39.227067 2583 scope.go:117] "RemoveContainer" containerID="b10995e185b398211fed021fe852d4a1d7c64981620999aea5fc4b5792660016" Sep 11 14:58:39.235384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:39.235367 2583 scope.go:117] "RemoveContainer" containerID="b10995e185b398211fed021fe852d4a1d7c64981620999aea5fc4b5792660016" Sep 11 14:58:39.235618 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:58:39.235603 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b10995e185b398211fed021fe852d4a1d7c64981620999aea5fc4b5792660016\": container with ID starting with b10995e185b398211fed021fe852d4a1d7c64981620999aea5fc4b5792660016 not found: ID does not exist" containerID="b10995e185b398211fed021fe852d4a1d7c64981620999aea5fc4b5792660016" Sep 11 14:58:39.235666 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:39.235624 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b10995e185b398211fed021fe852d4a1d7c64981620999aea5fc4b5792660016"} err="failed to get container status \"b10995e185b398211fed021fe852d4a1d7c64981620999aea5fc4b5792660016\": rpc error: code = NotFound desc = could not find container \"b10995e185b398211fed021fe852d4a1d7c64981620999aea5fc4b5792660016\": container with ID starting with b10995e185b398211fed021fe852d4a1d7c64981620999aea5fc4b5792660016 not found: ID does not exist" Sep 11 14:58:39.243477 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:39.243456 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94"] Sep 11 14:58:39.247339 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:39.247312 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-577852b1-c559688c4-4vb94"] Sep 11 14:58:39.257202 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:39.257181 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc"] Sep 11 14:58:39.263572 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:39.263551 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-577852b1-6585fb4f97-rhqsc"] Sep 11 14:58:39.331973 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:39.331939 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30a4d210-abcb-4372-8c3c-8610069ebe44" path="/var/lib/kubelet/pods/30a4d210-abcb-4372-8c3c-8610069ebe44/volumes" Sep 11 14:58:39.332356 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:39.332338 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb076503-cce1-429c-b2b0-d5d581ea4c10" path="/var/lib/kubelet/pods/bb076503-cce1-429c-b2b0-d5d581ea4c10/volumes" Sep 11 14:58:39.332754 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:39.332737 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2ef38a9-fa6c-4d00-9793-cd877453a927" path="/var/lib/kubelet/pods/c2ef38a9-fa6c-4d00-9793-cd877453a927/volumes" Sep 11 14:58:45.673802 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.673767 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7"] Sep 11 14:58:45.674323 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.674306 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb076503-cce1-429c-b2b0-d5d581ea4c10" containerName="revoke-keys" Sep 11 14:58:45.674373 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.674327 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb076503-cce1-429c-b2b0-d5d581ea4c10" containerName="revoke-keys" Sep 11 14:58:45.674373 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.674345 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2ef38a9-fa6c-4d00-9793-cd877453a927" containerName="payload-processing" Sep 11 14:58:45.674373 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.674351 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2ef38a9-fa6c-4d00-9793-cd877453a927" containerName="payload-processing" Sep 11 14:58:45.674373 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.674368 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30a4d210-abcb-4372-8c3c-8610069ebe44" containerName="payload-pre-processing" Sep 11 14:58:45.674373 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.674373 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="30a4d210-abcb-4372-8c3c-8610069ebe44" containerName="payload-pre-processing" Sep 11 14:58:45.674535 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.674434 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="30a4d210-abcb-4372-8c3c-8610069ebe44" containerName="payload-pre-processing" Sep 11 14:58:45.674535 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.674443 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2ef38a9-fa6c-4d00-9793-cd877453a927" containerName="payload-processing" Sep 11 14:58:45.674535 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.674450 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb076503-cce1-429c-b2b0-d5d581ea4c10" containerName="revoke-keys" Sep 11 14:58:45.679065 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.679046 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" Sep 11 14:58:45.681511 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.681493 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e7ca56\"/\"kube-root-ca.crt\"" Sep 11 14:58:45.682325 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.682304 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e7ca56\"/\"distinct-w0-e7ca56-kserve-self-signed-certs\"" Sep 11 14:58:45.682434 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.682343 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e7ca56\"/\"default-dockercfg-rhxhm\"" Sep 11 14:58:45.682434 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.682419 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e7ca56\"/\"openshift-service-ca.crt\"" Sep 11 14:58:45.696021 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.696002 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7"] Sep 11 14:58:45.837837 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.837806 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5v2r\" (UniqueName: \"kubernetes.io/projected/c1c39a2a-fe09-4b71-b448-03581b6975ad-kube-api-access-l5v2r\") pod \"distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7\" (UID: \"c1c39a2a-fe09-4b71-b448-03581b6975ad\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" Sep 11 14:58:45.838040 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.837853 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c39a2a-fe09-4b71-b448-03581b6975ad-model-cache\") pod \"distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7\" (UID: \"c1c39a2a-fe09-4b71-b448-03581b6975ad\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" Sep 11 14:58:45.838040 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.837923 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c39a2a-fe09-4b71-b448-03581b6975ad-home\") pod \"distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7\" (UID: \"c1c39a2a-fe09-4b71-b448-03581b6975ad\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" Sep 11 14:58:45.838040 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.837981 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c39a2a-fe09-4b71-b448-03581b6975ad-dshm\") pod \"distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7\" (UID: \"c1c39a2a-fe09-4b71-b448-03581b6975ad\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" Sep 11 14:58:45.838040 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.838001 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c39a2a-fe09-4b71-b448-03581b6975ad-tmp-dir\") pod \"distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7\" (UID: \"c1c39a2a-fe09-4b71-b448-03581b6975ad\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" Sep 11 14:58:45.838245 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.838058 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c39a2a-fe09-4b71-b448-03581b6975ad-tls-certs\") pod \"distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7\" (UID: \"c1c39a2a-fe09-4b71-b448-03581b6975ad\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" Sep 11 14:58:45.939516 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.939443 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c39a2a-fe09-4b71-b448-03581b6975ad-dshm\") pod \"distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7\" (UID: \"c1c39a2a-fe09-4b71-b448-03581b6975ad\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" Sep 11 14:58:45.939516 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.939481 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c39a2a-fe09-4b71-b448-03581b6975ad-tmp-dir\") pod \"distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7\" (UID: \"c1c39a2a-fe09-4b71-b448-03581b6975ad\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" Sep 11 14:58:45.939701 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.939595 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c39a2a-fe09-4b71-b448-03581b6975ad-tls-certs\") pod \"distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7\" (UID: \"c1c39a2a-fe09-4b71-b448-03581b6975ad\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" Sep 11 14:58:45.939701 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.939668 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5v2r\" (UniqueName: \"kubernetes.io/projected/c1c39a2a-fe09-4b71-b448-03581b6975ad-kube-api-access-l5v2r\") pod \"distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7\" (UID: \"c1c39a2a-fe09-4b71-b448-03581b6975ad\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" Sep 11 14:58:45.939832 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.939722 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c39a2a-fe09-4b71-b448-03581b6975ad-model-cache\") pod \"distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7\" (UID: \"c1c39a2a-fe09-4b71-b448-03581b6975ad\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" Sep 11 14:58:45.939832 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.939762 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c39a2a-fe09-4b71-b448-03581b6975ad-home\") pod \"distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7\" (UID: \"c1c39a2a-fe09-4b71-b448-03581b6975ad\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" Sep 11 14:58:45.939928 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.939873 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c39a2a-fe09-4b71-b448-03581b6975ad-tmp-dir\") pod \"distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7\" (UID: \"c1c39a2a-fe09-4b71-b448-03581b6975ad\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" Sep 11 14:58:45.940084 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.940061 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c39a2a-fe09-4b71-b448-03581b6975ad-model-cache\") pod \"distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7\" (UID: \"c1c39a2a-fe09-4b71-b448-03581b6975ad\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" Sep 11 14:58:45.940138 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.940067 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c39a2a-fe09-4b71-b448-03581b6975ad-home\") pod \"distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7\" (UID: \"c1c39a2a-fe09-4b71-b448-03581b6975ad\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" Sep 11 14:58:45.941793 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.941767 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c39a2a-fe09-4b71-b448-03581b6975ad-dshm\") pod \"distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7\" (UID: \"c1c39a2a-fe09-4b71-b448-03581b6975ad\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" Sep 11 14:58:45.941956 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.941937 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c39a2a-fe09-4b71-b448-03581b6975ad-tls-certs\") pod \"distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7\" (UID: \"c1c39a2a-fe09-4b71-b448-03581b6975ad\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" Sep 11 14:58:45.947898 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.947880 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5v2r\" (UniqueName: \"kubernetes.io/projected/c1c39a2a-fe09-4b71-b448-03581b6975ad-kube-api-access-l5v2r\") pod \"distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7\" (UID: \"c1c39a2a-fe09-4b71-b448-03581b6975ad\") " pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" Sep 11 14:58:45.988864 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:45.988836 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" Sep 11 14:58:46.291856 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:46.291821 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7"] Sep 11 14:58:46.293077 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:58:46.293045 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1c39a2a_fe09_4b71_b448_03581b6975ad.slice/crio-b364afef5d9a174786f16cd3ec731e27b247eef4893725b59488bc287f427817 WatchSource:0}: Error finding container b364afef5d9a174786f16cd3ec731e27b247eef4893725b59488bc287f427817: Status 404 returned error can't find the container with id b364afef5d9a174786f16cd3ec731e27b247eef4893725b59488bc287f427817 Sep 11 14:58:47.269602 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:47.269561 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" event={"ID":"c1c39a2a-fe09-4b71-b448-03581b6975ad","Type":"ContainerStarted","Data":"487c49604709e6ecd963de04f94975313bd71d7eec3c8d38d4d7e2dd50cdec22"} Sep 11 14:58:47.270025 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:47.269612 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" event={"ID":"c1c39a2a-fe09-4b71-b448-03581b6975ad","Type":"ContainerStarted","Data":"b364afef5d9a174786f16cd3ec731e27b247eef4893725b59488bc287f427817"} Sep 11 14:58:47.289887 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:47.289835 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" podStartSLOduration=2.289819615 podStartE2EDuration="2.289819615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:58:47.286970945 +0000 UTC m=+1118.530956606" watchObservedRunningTime="2026-09-11 14:58:47.289819615 +0000 UTC m=+1118.533805274" Sep 11 14:58:49.584348 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:49.584309 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87d086-kkp5l"] Sep 11 14:58:49.588243 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:49.588220 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87d086-kkp5l" Sep 11 14:58:49.590755 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:49.590736 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:58:49.590865 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:49.590777 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nks8w\"" Sep 11 14:58:49.596870 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:49.596336 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87d086-kkp5l"] Sep 11 14:58:49.675184 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:49.675150 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b35c1a-9046-4e61-a696-07cdb912886e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-87d086-kkp5l\" (UID: \"d8b35c1a-9046-4e61-a696-07cdb912886e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87d086-kkp5l" Sep 11 14:58:49.776000 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:49.775961 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b35c1a-9046-4e61-a696-07cdb912886e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-87d086-kkp5l\" (UID: \"d8b35c1a-9046-4e61-a696-07cdb912886e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87d086-kkp5l" Sep 11 14:58:49.776620 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:49.776598 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b35c1a-9046-4e61-a696-07cdb912886e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-87d086-kkp5l\" (UID: \"d8b35c1a-9046-4e61-a696-07cdb912886e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87d086-kkp5l" Sep 11 14:58:49.899999 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:49.899967 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87d086-kkp5l" Sep 11 14:58:50.174687 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:50.174656 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87d086-kkp5l"] Sep 11 14:58:50.177394 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:58:50.177366 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b35c1a_9046_4e61_a696_07cdb912886e.slice/crio-fac28892db0efc2b1e10eee10e7f3a4a887ca76ffdd1b9b7bbd7ba5898aea8aa WatchSource:0}: Error finding container fac28892db0efc2b1e10eee10e7f3a4a887ca76ffdd1b9b7bbd7ba5898aea8aa: Status 404 returned error can't find the container with id fac28892db0efc2b1e10eee10e7f3a4a887ca76ffdd1b9b7bbd7ba5898aea8aa Sep 11 14:58:50.290575 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:50.290540 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87d086-kkp5l" event={"ID":"d8b35c1a-9046-4e61-a696-07cdb912886e","Type":"ContainerStarted","Data":"fac28892db0efc2b1e10eee10e7f3a4a887ca76ffdd1b9b7bbd7ba5898aea8aa"} Sep 11 14:58:51.297964 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:51.297927 2583 generic.go:356] "Generic (PLEG): container finished" podID="d8b35c1a-9046-4e61-a696-07cdb912886e" containerID="44c57f6dc3086237ebe67590fd79b9dc17faf79d22207ddbe40ed577f0ad58df" exitCode=0 Sep 11 14:58:51.298361 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:51.298023 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87d086-kkp5l" event={"ID":"d8b35c1a-9046-4e61-a696-07cdb912886e","Type":"ContainerDied","Data":"44c57f6dc3086237ebe67590fd79b9dc17faf79d22207ddbe40ed577f0ad58df"} Sep 11 14:58:52.445801 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:52.445772 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87d086-kkp5l" Sep 11 14:58:52.501566 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:52.501532 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b35c1a-9046-4e61-a696-07cdb912886e-maas-api-service-ca\") pod \"d8b35c1a-9046-4e61-a696-07cdb912886e\" (UID: \"d8b35c1a-9046-4e61-a696-07cdb912886e\") " Sep 11 14:58:52.501913 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:52.501892 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b35c1a-9046-4e61-a696-07cdb912886e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d8b35c1a-9046-4e61-a696-07cdb912886e" (UID: "d8b35c1a-9046-4e61-a696-07cdb912886e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:58:52.602892 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:52.602816 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b35c1a-9046-4e61-a696-07cdb912886e-maas-api-service-ca\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:58:53.311129 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:53.311098 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87d086-kkp5l" Sep 11 14:58:53.311342 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:53.311139 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87d086-kkp5l" event={"ID":"d8b35c1a-9046-4e61-a696-07cdb912886e","Type":"ContainerDied","Data":"fac28892db0efc2b1e10eee10e7f3a4a887ca76ffdd1b9b7bbd7ba5898aea8aa"} Sep 11 14:58:53.311342 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:53.311186 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fac28892db0efc2b1e10eee10e7f3a4a887ca76ffdd1b9b7bbd7ba5898aea8aa" Sep 11 14:58:55.989202 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:55.989171 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" Sep 11 14:58:55.989618 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:55.989429 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" Sep 11 14:58:56.005785 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:56.005758 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" Sep 11 14:58:56.341271 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:58:56.341189 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" Sep 11 14:59:00.194418 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:00.194375 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl"] Sep 11 14:59:00.194926 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:00.194647 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl" podUID="5005ad13-1c15-4f0a-8787-48b604de27f2" containerName="payload-processing" containerID="cri-o://b347c665705ca1973169f72b231f61fe9bf0efd4c0d9027202ae04eae17c2e5c" gracePeriod=30 Sep 11 14:59:00.198307 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:00.197845 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx"] Sep 11 14:59:00.198718 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:00.198673 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx" podUID="fab6801d-4d6a-4fb0-81b7-822f7192051e" containerName="payload-pre-processing" containerID="cri-o://93c42ac95a2259ce4e798dcaff37638add54511fb0f7267b0a92cd260b59826b" gracePeriod=30 Sep 11 14:59:00.354978 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:00.354936 2583 generic.go:356] "Generic (PLEG): container finished" podID="5005ad13-1c15-4f0a-8787-48b604de27f2" containerID="b347c665705ca1973169f72b231f61fe9bf0efd4c0d9027202ae04eae17c2e5c" exitCode=0 Sep 11 14:59:00.355169 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:00.355019 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl" event={"ID":"5005ad13-1c15-4f0a-8787-48b604de27f2","Type":"ContainerDied","Data":"b347c665705ca1973169f72b231f61fe9bf0efd4c0d9027202ae04eae17c2e5c"} Sep 11 14:59:00.356837 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:00.356812 2583 generic.go:356] "Generic (PLEG): container finished" podID="fab6801d-4d6a-4fb0-81b7-822f7192051e" containerID="93c42ac95a2259ce4e798dcaff37638add54511fb0f7267b0a92cd260b59826b" exitCode=0 Sep 11 14:59:00.356933 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:00.356882 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx" event={"ID":"fab6801d-4d6a-4fb0-81b7-822f7192051e","Type":"ContainerDied","Data":"93c42ac95a2259ce4e798dcaff37638add54511fb0f7267b0a92cd260b59826b"} Sep 11 14:59:00.413208 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:00.413176 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87d086-kkp5l"] Sep 11 14:59:00.415852 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:00.415828 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87d086-kkp5l"] Sep 11 14:59:00.479492 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:00.479468 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl" Sep 11 14:59:00.482350 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:00.482332 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx" Sep 11 14:59:00.580251 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:00.580224 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fab6801d-4d6a-4fb0-81b7-822f7192051e-plugins-config-volume\") pod \"fab6801d-4d6a-4fb0-81b7-822f7192051e\" (UID: \"fab6801d-4d6a-4fb0-81b7-822f7192051e\") " Sep 11 14:59:00.580251 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:00.580258 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5005ad13-1c15-4f0a-8787-48b604de27f2-plugins-config-volume\") pod \"5005ad13-1c15-4f0a-8787-48b604de27f2\" (UID: \"5005ad13-1c15-4f0a-8787-48b604de27f2\") " Sep 11 14:59:00.580504 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:00.580290 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rs7k\" (UniqueName: \"kubernetes.io/projected/fab6801d-4d6a-4fb0-81b7-822f7192051e-kube-api-access-6rs7k\") pod \"fab6801d-4d6a-4fb0-81b7-822f7192051e\" (UID: \"fab6801d-4d6a-4fb0-81b7-822f7192051e\") " Sep 11 14:59:00.580504 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:00.580353 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4d9q4\" (UniqueName: \"kubernetes.io/projected/5005ad13-1c15-4f0a-8787-48b604de27f2-kube-api-access-4d9q4\") pod \"5005ad13-1c15-4f0a-8787-48b604de27f2\" (UID: \"5005ad13-1c15-4f0a-8787-48b604de27f2\") " Sep 11 14:59:00.580616 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:00.580592 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fab6801d-4d6a-4fb0-81b7-822f7192051e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fab6801d-4d6a-4fb0-81b7-822f7192051e" (UID: "fab6801d-4d6a-4fb0-81b7-822f7192051e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:59:00.580671 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:00.580592 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5005ad13-1c15-4f0a-8787-48b604de27f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5005ad13-1c15-4f0a-8787-48b604de27f2" (UID: "5005ad13-1c15-4f0a-8787-48b604de27f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:59:00.582497 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:00.582476 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab6801d-4d6a-4fb0-81b7-822f7192051e-kube-api-access-6rs7k" (OuterVolumeSpecName: "kube-api-access-6rs7k") pod "fab6801d-4d6a-4fb0-81b7-822f7192051e" (UID: "fab6801d-4d6a-4fb0-81b7-822f7192051e"). InnerVolumeSpecName "kube-api-access-6rs7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:59:00.582497 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:00.582485 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5005ad13-1c15-4f0a-8787-48b604de27f2-kube-api-access-4d9q4" (OuterVolumeSpecName: "kube-api-access-4d9q4") pod "5005ad13-1c15-4f0a-8787-48b604de27f2" (UID: "5005ad13-1c15-4f0a-8787-48b604de27f2"). InnerVolumeSpecName "kube-api-access-4d9q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:59:00.681929 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:00.681902 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rs7k\" (UniqueName: \"kubernetes.io/projected/fab6801d-4d6a-4fb0-81b7-822f7192051e-kube-api-access-6rs7k\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:59:00.681929 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:00.681926 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d9q4\" (UniqueName: \"kubernetes.io/projected/5005ad13-1c15-4f0a-8787-48b604de27f2-kube-api-access-4d9q4\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:59:00.682089 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:00.681936 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fab6801d-4d6a-4fb0-81b7-822f7192051e-plugins-config-volume\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:59:00.682089 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:00.681946 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5005ad13-1c15-4f0a-8787-48b604de27f2-plugins-config-volume\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:59:01.331249 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:01.331218 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8b35c1a-9046-4e61-a696-07cdb912886e" path="/var/lib/kubelet/pods/d8b35c1a-9046-4e61-a696-07cdb912886e/volumes" Sep 11 14:59:01.363885 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:01.363861 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl" Sep 11 14:59:01.364030 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:01.363859 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl" event={"ID":"5005ad13-1c15-4f0a-8787-48b604de27f2","Type":"ContainerDied","Data":"0bbd441a118e52cc122a18f8fbab247c64faf188e5f0073cd5ef85be1ca40e28"} Sep 11 14:59:01.364030 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:01.363995 2583 scope.go:117] "RemoveContainer" containerID="b347c665705ca1973169f72b231f61fe9bf0efd4c0d9027202ae04eae17c2e5c" Sep 11 14:59:01.368126 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:01.368101 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx" Sep 11 14:59:01.368358 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:01.368140 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx" event={"ID":"fab6801d-4d6a-4fb0-81b7-822f7192051e","Type":"ContainerDied","Data":"0b907c919cdf5a89facec342e29325a99a3adefa4845fb2296d01696ff629890"} Sep 11 14:59:01.375780 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:01.375760 2583 scope.go:117] "RemoveContainer" containerID="93c42ac95a2259ce4e798dcaff37638add54511fb0f7267b0a92cd260b59826b" Sep 11 14:59:01.381430 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:01.381409 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl"] Sep 11 14:59:01.385559 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:01.385541 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-87d086-76c8ddffb5-5rrbl"] Sep 11 14:59:01.395951 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:01.395930 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx"] Sep 11 14:59:01.402002 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:01.401985 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-87d086-6598797bf4-ccnqx"] Sep 11 14:59:03.331423 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:03.331382 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5005ad13-1c15-4f0a-8787-48b604de27f2" path="/var/lib/kubelet/pods/5005ad13-1c15-4f0a-8787-48b604de27f2/volumes" Sep 11 14:59:03.331990 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:03.331969 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fab6801d-4d6a-4fb0-81b7-822f7192051e" path="/var/lib/kubelet/pods/fab6801d-4d6a-4fb0-81b7-822f7192051e/volumes" Sep 11 14:59:10.802626 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:10.802589 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w"] Sep 11 14:59:10.803086 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:10.803007 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5005ad13-1c15-4f0a-8787-48b604de27f2" containerName="payload-processing" Sep 11 14:59:10.803086 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:10.803024 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5005ad13-1c15-4f0a-8787-48b604de27f2" containerName="payload-processing" Sep 11 14:59:10.803086 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:10.803043 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8b35c1a-9046-4e61-a696-07cdb912886e" containerName="revoke-keys" Sep 11 14:59:10.803086 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:10.803049 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8b35c1a-9046-4e61-a696-07cdb912886e" containerName="revoke-keys" Sep 11 14:59:10.803086 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:10.803056 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fab6801d-4d6a-4fb0-81b7-822f7192051e" containerName="payload-pre-processing" Sep 11 14:59:10.803086 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:10.803062 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fab6801d-4d6a-4fb0-81b7-822f7192051e" containerName="payload-pre-processing" Sep 11 14:59:10.803402 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:10.803147 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5005ad13-1c15-4f0a-8787-48b604de27f2" containerName="payload-processing" Sep 11 14:59:10.803402 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:10.803163 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8b35c1a-9046-4e61-a696-07cdb912886e" containerName="revoke-keys" Sep 11 14:59:10.803402 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:10.803171 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fab6801d-4d6a-4fb0-81b7-822f7192051e" containerName="payload-pre-processing" Sep 11 14:59:10.807791 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:10.807772 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w" Sep 11 14:59:10.809980 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:10.809959 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-c4e39b\"" Sep 11 14:59:10.810100 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:10.810065 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-c4e39b-dockercfg-rm65r\"" Sep 11 14:59:10.819595 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:10.819573 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w"] Sep 11 14:59:10.887107 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:10.887073 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjjj8\" (UniqueName: \"kubernetes.io/projected/8c7e3927-4576-4427-8613-496e7599e40e-kube-api-access-mjjj8\") pod \"payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w\" (UID: \"8c7e3927-4576-4427-8613-496e7599e40e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w" Sep 11 14:59:10.887255 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:10.887156 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c7e3927-4576-4427-8613-496e7599e40e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w\" (UID: \"8c7e3927-4576-4427-8613-496e7599e40e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w" Sep 11 14:59:10.988542 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:10.988502 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c7e3927-4576-4427-8613-496e7599e40e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w\" (UID: \"8c7e3927-4576-4427-8613-496e7599e40e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w" Sep 11 14:59:10.988755 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:10.988602 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjjj8\" (UniqueName: \"kubernetes.io/projected/8c7e3927-4576-4427-8613-496e7599e40e-kube-api-access-mjjj8\") pod \"payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w\" (UID: \"8c7e3927-4576-4427-8613-496e7599e40e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w" Sep 11 14:59:10.989380 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:10.989354 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c7e3927-4576-4427-8613-496e7599e40e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w\" (UID: \"8c7e3927-4576-4427-8613-496e7599e40e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w" Sep 11 14:59:11.006754 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:11.006688 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjjj8\" (UniqueName: \"kubernetes.io/projected/8c7e3927-4576-4427-8613-496e7599e40e-kube-api-access-mjjj8\") pod \"payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w\" (UID: \"8c7e3927-4576-4427-8613-496e7599e40e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w" Sep 11 14:59:11.118777 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:11.118672 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w" Sep 11 14:59:11.402413 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:11.402385 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w"] Sep 11 14:59:11.403910 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:59:11.403882 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c7e3927_4576_4427_8613_496e7599e40e.slice/crio-f41ad68eb03de7baa3e8efbeb40bd1822ef9fee6dfdaa0327eca71077538d0a1 WatchSource:0}: Error finding container f41ad68eb03de7baa3e8efbeb40bd1822ef9fee6dfdaa0327eca71077538d0a1: Status 404 returned error can't find the container with id f41ad68eb03de7baa3e8efbeb40bd1822ef9fee6dfdaa0327eca71077538d0a1 Sep 11 14:59:11.434433 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:11.434396 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w" event={"ID":"8c7e3927-4576-4427-8613-496e7599e40e","Type":"ContainerStarted","Data":"f41ad68eb03de7baa3e8efbeb40bd1822ef9fee6dfdaa0327eca71077538d0a1"} Sep 11 14:59:12.443728 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:12.443664 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w" event={"ID":"8c7e3927-4576-4427-8613-496e7599e40e","Type":"ContainerStarted","Data":"4c901f79ec65d582acbb43e46929a800bec91402f53356d848cc01eb5e936364"} Sep 11 14:59:12.444204 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:12.443875 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w" Sep 11 14:59:12.463368 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:12.463310 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w" podStartSLOduration=2.463290329 podStartE2EDuration="2.463290329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:59:12.461831671 +0000 UTC m=+1143.705817332" watchObservedRunningTime="2026-09-11 14:59:12.463290329 +0000 UTC m=+1143.707275989" Sep 11 14:59:22.710172 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:22.710136 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-362f7a-hrbjq"] Sep 11 14:59:22.713937 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:22.713919 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-362f7a-hrbjq" Sep 11 14:59:22.716423 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:22.716402 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nks8w\"" Sep 11 14:59:22.716541 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:22.716408 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:59:22.721181 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:22.721159 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-362f7a-hrbjq"] Sep 11 14:59:22.803883 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:22.803839 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a990bbc0-7844-44ad-a0d8-ac7e038846d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-362f7a-hrbjq\" (UID: \"a990bbc0-7844-44ad-a0d8-ac7e038846d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-362f7a-hrbjq" Sep 11 14:59:22.904664 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:22.904628 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a990bbc0-7844-44ad-a0d8-ac7e038846d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-362f7a-hrbjq\" (UID: \"a990bbc0-7844-44ad-a0d8-ac7e038846d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-362f7a-hrbjq" Sep 11 14:59:22.905330 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:22.905310 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a990bbc0-7844-44ad-a0d8-ac7e038846d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-362f7a-hrbjq\" (UID: \"a990bbc0-7844-44ad-a0d8-ac7e038846d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-362f7a-hrbjq" Sep 11 14:59:23.027918 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:23.027836 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-362f7a-hrbjq" Sep 11 14:59:23.306575 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:23.306547 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-362f7a-hrbjq"] Sep 11 14:59:23.308458 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:59:23.308432 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda990bbc0_7844_44ad_a0d8_ac7e038846d5.slice/crio-6c32332c00aba055b97a565440939b8414696bac647d484cc4183fcf07697c31 WatchSource:0}: Error finding container 6c32332c00aba055b97a565440939b8414696bac647d484cc4183fcf07697c31: Status 404 returned error can't find the container with id 6c32332c00aba055b97a565440939b8414696bac647d484cc4183fcf07697c31 Sep 11 14:59:23.451811 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:23.451790 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w" Sep 11 14:59:23.512354 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:23.512320 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-362f7a-hrbjq" event={"ID":"a990bbc0-7844-44ad-a0d8-ac7e038846d5","Type":"ContainerStarted","Data":"6c32332c00aba055b97a565440939b8414696bac647d484cc4183fcf07697c31"} Sep 11 14:59:24.519635 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:24.519603 2583 generic.go:356] "Generic (PLEG): container finished" podID="a990bbc0-7844-44ad-a0d8-ac7e038846d5" containerID="f9d2f99d0e7fd2731048cea2da615f5d5457840d92229991f31b40a8d1294de1" exitCode=0 Sep 11 14:59:24.520031 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:24.519653 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-362f7a-hrbjq" event={"ID":"a990bbc0-7844-44ad-a0d8-ac7e038846d5","Type":"ContainerDied","Data":"f9d2f99d0e7fd2731048cea2da615f5d5457840d92229991f31b40a8d1294de1"} Sep 11 14:59:25.663690 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:25.663667 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-362f7a-hrbjq" Sep 11 14:59:25.831313 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:25.831240 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a990bbc0-7844-44ad-a0d8-ac7e038846d5-maas-api-service-ca\") pod \"a990bbc0-7844-44ad-a0d8-ac7e038846d5\" (UID: \"a990bbc0-7844-44ad-a0d8-ac7e038846d5\") " Sep 11 14:59:25.831574 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:25.831552 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a990bbc0-7844-44ad-a0d8-ac7e038846d5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a990bbc0-7844-44ad-a0d8-ac7e038846d5" (UID: "a990bbc0-7844-44ad-a0d8-ac7e038846d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:59:25.932542 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:25.932515 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a990bbc0-7844-44ad-a0d8-ac7e038846d5-maas-api-service-ca\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:59:26.533244 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:26.533215 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-362f7a-hrbjq" Sep 11 14:59:26.533244 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:26.533232 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-362f7a-hrbjq" event={"ID":"a990bbc0-7844-44ad-a0d8-ac7e038846d5","Type":"ContainerDied","Data":"6c32332c00aba055b97a565440939b8414696bac647d484cc4183fcf07697c31"} Sep 11 14:59:26.533498 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:26.533281 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c32332c00aba055b97a565440939b8414696bac647d484cc4183fcf07697c31" Sep 11 14:59:32.197235 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:32.197156 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl"] Sep 11 14:59:32.197685 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:32.197590 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a990bbc0-7844-44ad-a0d8-ac7e038846d5" containerName="revoke-keys" Sep 11 14:59:32.197685 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:32.197603 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a990bbc0-7844-44ad-a0d8-ac7e038846d5" containerName="revoke-keys" Sep 11 14:59:32.197685 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:32.197669 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a990bbc0-7844-44ad-a0d8-ac7e038846d5" containerName="revoke-keys" Sep 11 14:59:32.200904 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:32.200888 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl" Sep 11 14:59:32.203173 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:32.203145 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-c0e28e\"" Sep 11 14:59:32.218204 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:32.218173 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl"] Sep 11 14:59:32.226363 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:32.226335 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt"] Sep 11 14:59:32.230235 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:32.230218 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt" Sep 11 14:59:32.232896 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:32.232860 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-c0e28e-dockercfg-bjl5j\"" Sep 11 14:59:32.243495 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:32.243471 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt"] Sep 11 14:59:32.290339 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:32.290300 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f624r\" (UniqueName: \"kubernetes.io/projected/7f6eb126-f22c-4ca2-a056-5e160c0fc3cf-kube-api-access-f624r\") pod \"payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl\" (UID: \"7f6eb126-f22c-4ca2-a056-5e160c0fc3cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl" Sep 11 14:59:32.290339 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:32.290343 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc855\" (UniqueName: \"kubernetes.io/projected/b4772d28-a1c5-454b-ace5-903df84a4728-kube-api-access-xc855\") pod \"payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt\" (UID: \"b4772d28-a1c5-454b-ace5-903df84a4728\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt" Sep 11 14:59:32.290561 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:32.290379 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4772d28-a1c5-454b-ace5-903df84a4728-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt\" (UID: \"b4772d28-a1c5-454b-ace5-903df84a4728\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt" Sep 11 14:59:32.290561 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:32.290444 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f6eb126-f22c-4ca2-a056-5e160c0fc3cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl\" (UID: \"7f6eb126-f22c-4ca2-a056-5e160c0fc3cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl" Sep 11 14:59:32.391214 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:32.391176 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f624r\" (UniqueName: \"kubernetes.io/projected/7f6eb126-f22c-4ca2-a056-5e160c0fc3cf-kube-api-access-f624r\") pod \"payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl\" (UID: \"7f6eb126-f22c-4ca2-a056-5e160c0fc3cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl" Sep 11 14:59:32.391366 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:32.391222 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc855\" (UniqueName: \"kubernetes.io/projected/b4772d28-a1c5-454b-ace5-903df84a4728-kube-api-access-xc855\") pod \"payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt\" (UID: \"b4772d28-a1c5-454b-ace5-903df84a4728\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt" Sep 11 14:59:32.391366 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:32.391316 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4772d28-a1c5-454b-ace5-903df84a4728-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt\" (UID: \"b4772d28-a1c5-454b-ace5-903df84a4728\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt" Sep 11 14:59:32.391473 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:32.391392 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f6eb126-f22c-4ca2-a056-5e160c0fc3cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl\" (UID: \"7f6eb126-f22c-4ca2-a056-5e160c0fc3cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl" Sep 11 14:59:32.392047 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:32.392021 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4772d28-a1c5-454b-ace5-903df84a4728-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt\" (UID: \"b4772d28-a1c5-454b-ace5-903df84a4728\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt" Sep 11 14:59:32.392189 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:32.392169 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f6eb126-f22c-4ca2-a056-5e160c0fc3cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl\" (UID: \"7f6eb126-f22c-4ca2-a056-5e160c0fc3cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl" Sep 11 14:59:32.399878 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:32.399858 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f624r\" (UniqueName: \"kubernetes.io/projected/7f6eb126-f22c-4ca2-a056-5e160c0fc3cf-kube-api-access-f624r\") pod \"payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl\" (UID: \"7f6eb126-f22c-4ca2-a056-5e160c0fc3cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl" Sep 11 14:59:32.399967 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:32.399904 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc855\" (UniqueName: \"kubernetes.io/projected/b4772d28-a1c5-454b-ace5-903df84a4728-kube-api-access-xc855\") pod \"payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt\" (UID: \"b4772d28-a1c5-454b-ace5-903df84a4728\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt" Sep 11 14:59:32.513508 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:32.513453 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl" Sep 11 14:59:32.542786 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:32.542758 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt" Sep 11 14:59:32.798224 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:32.798198 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl"] Sep 11 14:59:32.799362 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:59:32.799341 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f6eb126_f22c_4ca2_a056_5e160c0fc3cf.slice/crio-b82dd710af0da597d7da5033c4d474ad62ee02c8d4cada093df6f4c8b0427df7 WatchSource:0}: Error finding container b82dd710af0da597d7da5033c4d474ad62ee02c8d4cada093df6f4c8b0427df7: Status 404 returned error can't find the container with id b82dd710af0da597d7da5033c4d474ad62ee02c8d4cada093df6f4c8b0427df7 Sep 11 14:59:32.845065 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:59:32.845041 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4772d28_a1c5_454b_ace5_903df84a4728.slice/crio-012053e3ce886e1dad535165853368ea651d6087ba9dca02807b754b49c36184 WatchSource:0}: Error finding container 012053e3ce886e1dad535165853368ea651d6087ba9dca02807b754b49c36184: Status 404 returned error can't find the container with id 012053e3ce886e1dad535165853368ea651d6087ba9dca02807b754b49c36184 Sep 11 14:59:32.845188 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:32.845086 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt"] Sep 11 14:59:33.590931 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:33.590893 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl" event={"ID":"7f6eb126-f22c-4ca2-a056-5e160c0fc3cf","Type":"ContainerStarted","Data":"a3f56e266b1138defd0af1eb78e946c930d736311995f4fdd1d318f08a38e094"} Sep 11 14:59:33.592044 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:33.592014 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl" event={"ID":"7f6eb126-f22c-4ca2-a056-5e160c0fc3cf","Type":"ContainerStarted","Data":"b82dd710af0da597d7da5033c4d474ad62ee02c8d4cada093df6f4c8b0427df7"} Sep 11 14:59:33.592424 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:33.592324 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl" Sep 11 14:59:33.594432 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:33.594403 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt" event={"ID":"b4772d28-a1c5-454b-ace5-903df84a4728","Type":"ContainerStarted","Data":"753f61ca23b0ba4bf741fce42e96e868218abaef4409ac53e5100f34f7d8f265"} Sep 11 14:59:33.594541 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:33.594455 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt" event={"ID":"b4772d28-a1c5-454b-ace5-903df84a4728","Type":"ContainerStarted","Data":"012053e3ce886e1dad535165853368ea651d6087ba9dca02807b754b49c36184"} Sep 11 14:59:33.594621 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:33.594584 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt" Sep 11 14:59:33.611226 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:33.611186 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl" podStartSLOduration=1.6111729590000001 podStartE2EDuration="1.611172959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:59:33.609287353 +0000 UTC m=+1164.853273013" watchObservedRunningTime="2026-09-11 14:59:33.611172959 +0000 UTC m=+1164.855158618" Sep 11 14:59:33.631321 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:33.631273 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt" podStartSLOduration=1.6312588099999998 podStartE2EDuration="1.63125881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:59:33.629201182 +0000 UTC m=+1164.873186841" watchObservedRunningTime="2026-09-11 14:59:33.63125881 +0000 UTC m=+1164.875244469" Sep 11 14:59:34.215989 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.215955 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc"] Sep 11 14:59:34.216227 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.216195 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc" podUID="94105113-4ed4-4985-86bf-911aa03ba7fd" containerName="payload-processing" containerID="cri-o://cadef8c5da8af9a33a19d830785942bd66f7aab86cd37f333f5821359ab574dd" gracePeriod=30 Sep 11 14:59:34.239004 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.232318 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj"] Sep 11 14:59:34.239004 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.232523 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj" podUID="8e861242-c72a-48df-a49f-654e107cbd3c" containerName="payload-pre-processing" containerID="cri-o://20889530de713cc4f662eff0f35570e6a5fa2bb668d19c038b38a0bb0cf7e4ae" gracePeriod=30 Sep 11 14:59:34.494224 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.494136 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-362f7a-hrbjq"] Sep 11 14:59:34.500082 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.500055 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-362f7a-hrbjq"] Sep 11 14:59:34.534672 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.534650 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc" Sep 11 14:59:34.537743 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.537724 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj" Sep 11 14:59:34.603320 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.603289 2583 generic.go:356] "Generic (PLEG): container finished" podID="8e861242-c72a-48df-a49f-654e107cbd3c" containerID="20889530de713cc4f662eff0f35570e6a5fa2bb668d19c038b38a0bb0cf7e4ae" exitCode=0 Sep 11 14:59:34.603760 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.603333 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj" Sep 11 14:59:34.603760 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.603380 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj" event={"ID":"8e861242-c72a-48df-a49f-654e107cbd3c","Type":"ContainerDied","Data":"20889530de713cc4f662eff0f35570e6a5fa2bb668d19c038b38a0bb0cf7e4ae"} Sep 11 14:59:34.603760 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.603450 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj" event={"ID":"8e861242-c72a-48df-a49f-654e107cbd3c","Type":"ContainerDied","Data":"4e57262ab8320f7aa174a142c056489af126af8d2e52b075d1c22057200a4272"} Sep 11 14:59:34.603760 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.603480 2583 scope.go:117] "RemoveContainer" containerID="20889530de713cc4f662eff0f35570e6a5fa2bb668d19c038b38a0bb0cf7e4ae" Sep 11 14:59:34.605031 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.605007 2583 generic.go:356] "Generic (PLEG): container finished" podID="94105113-4ed4-4985-86bf-911aa03ba7fd" containerID="cadef8c5da8af9a33a19d830785942bd66f7aab86cd37f333f5821359ab574dd" exitCode=0 Sep 11 14:59:34.605133 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.605032 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc" Sep 11 14:59:34.605133 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.605108 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc" event={"ID":"94105113-4ed4-4985-86bf-911aa03ba7fd","Type":"ContainerDied","Data":"cadef8c5da8af9a33a19d830785942bd66f7aab86cd37f333f5821359ab574dd"} Sep 11 14:59:34.605437 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.605416 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc" event={"ID":"94105113-4ed4-4985-86bf-911aa03ba7fd","Type":"ContainerDied","Data":"b55a272996f48f3abc0438c803da081f42495d49f850d5ddc10a8ec54faa0189"} Sep 11 14:59:34.613402 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.613381 2583 scope.go:117] "RemoveContainer" containerID="20889530de713cc4f662eff0f35570e6a5fa2bb668d19c038b38a0bb0cf7e4ae" Sep 11 14:59:34.613673 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:59:34.613655 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20889530de713cc4f662eff0f35570e6a5fa2bb668d19c038b38a0bb0cf7e4ae\": container with ID starting with 20889530de713cc4f662eff0f35570e6a5fa2bb668d19c038b38a0bb0cf7e4ae not found: ID does not exist" containerID="20889530de713cc4f662eff0f35570e6a5fa2bb668d19c038b38a0bb0cf7e4ae" Sep 11 14:59:34.613743 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.613681 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20889530de713cc4f662eff0f35570e6a5fa2bb668d19c038b38a0bb0cf7e4ae"} err="failed to get container status \"20889530de713cc4f662eff0f35570e6a5fa2bb668d19c038b38a0bb0cf7e4ae\": rpc error: code = NotFound desc = could not find container \"20889530de713cc4f662eff0f35570e6a5fa2bb668d19c038b38a0bb0cf7e4ae\": container with ID starting with 20889530de713cc4f662eff0f35570e6a5fa2bb668d19c038b38a0bb0cf7e4ae not found: ID does not exist" Sep 11 14:59:34.613743 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.613697 2583 scope.go:117] "RemoveContainer" containerID="cadef8c5da8af9a33a19d830785942bd66f7aab86cd37f333f5821359ab574dd" Sep 11 14:59:34.614366 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.614346 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e861242-c72a-48df-a49f-654e107cbd3c-plugins-config-volume\") pod \"8e861242-c72a-48df-a49f-654e107cbd3c\" (UID: \"8e861242-c72a-48df-a49f-654e107cbd3c\") " Sep 11 14:59:34.614465 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.614424 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hx9sx\" (UniqueName: \"kubernetes.io/projected/8e861242-c72a-48df-a49f-654e107cbd3c-kube-api-access-hx9sx\") pod \"8e861242-c72a-48df-a49f-654e107cbd3c\" (UID: \"8e861242-c72a-48df-a49f-654e107cbd3c\") " Sep 11 14:59:34.614522 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.614479 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94105113-4ed4-4985-86bf-911aa03ba7fd-plugins-config-volume\") pod \"94105113-4ed4-4985-86bf-911aa03ba7fd\" (UID: \"94105113-4ed4-4985-86bf-911aa03ba7fd\") " Sep 11 14:59:34.614580 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.614542 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gzk4\" (UniqueName: \"kubernetes.io/projected/94105113-4ed4-4985-86bf-911aa03ba7fd-kube-api-access-8gzk4\") pod \"94105113-4ed4-4985-86bf-911aa03ba7fd\" (UID: \"94105113-4ed4-4985-86bf-911aa03ba7fd\") " Sep 11 14:59:34.614742 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.614652 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e861242-c72a-48df-a49f-654e107cbd3c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8e861242-c72a-48df-a49f-654e107cbd3c" (UID: "8e861242-c72a-48df-a49f-654e107cbd3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:59:34.614927 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.614897 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e861242-c72a-48df-a49f-654e107cbd3c-plugins-config-volume\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:59:34.615021 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.614986 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94105113-4ed4-4985-86bf-911aa03ba7fd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "94105113-4ed4-4985-86bf-911aa03ba7fd" (UID: "94105113-4ed4-4985-86bf-911aa03ba7fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:59:34.616698 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.616677 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94105113-4ed4-4985-86bf-911aa03ba7fd-kube-api-access-8gzk4" (OuterVolumeSpecName: "kube-api-access-8gzk4") pod "94105113-4ed4-4985-86bf-911aa03ba7fd" (UID: "94105113-4ed4-4985-86bf-911aa03ba7fd"). InnerVolumeSpecName "kube-api-access-8gzk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:59:34.616810 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.616695 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e861242-c72a-48df-a49f-654e107cbd3c-kube-api-access-hx9sx" (OuterVolumeSpecName: "kube-api-access-hx9sx") pod "8e861242-c72a-48df-a49f-654e107cbd3c" (UID: "8e861242-c72a-48df-a49f-654e107cbd3c"). InnerVolumeSpecName "kube-api-access-hx9sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:59:34.632325 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.632304 2583 scope.go:117] "RemoveContainer" containerID="cadef8c5da8af9a33a19d830785942bd66f7aab86cd37f333f5821359ab574dd" Sep 11 14:59:34.632921 ip-10-0-135-39 kubenswrapper[2583]: E0911 14:59:34.632839 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cadef8c5da8af9a33a19d830785942bd66f7aab86cd37f333f5821359ab574dd\": container with ID starting with cadef8c5da8af9a33a19d830785942bd66f7aab86cd37f333f5821359ab574dd not found: ID does not exist" containerID="cadef8c5da8af9a33a19d830785942bd66f7aab86cd37f333f5821359ab574dd" Sep 11 14:59:34.632921 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.632875 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cadef8c5da8af9a33a19d830785942bd66f7aab86cd37f333f5821359ab574dd"} err="failed to get container status \"cadef8c5da8af9a33a19d830785942bd66f7aab86cd37f333f5821359ab574dd\": rpc error: code = NotFound desc = could not find container \"cadef8c5da8af9a33a19d830785942bd66f7aab86cd37f333f5821359ab574dd\": container with ID starting with cadef8c5da8af9a33a19d830785942bd66f7aab86cd37f333f5821359ab574dd not found: ID does not exist" Sep 11 14:59:34.715924 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.715897 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx9sx\" (UniqueName: \"kubernetes.io/projected/8e861242-c72a-48df-a49f-654e107cbd3c-kube-api-access-hx9sx\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:59:34.715924 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.715923 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94105113-4ed4-4985-86bf-911aa03ba7fd-plugins-config-volume\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:59:34.716092 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.715932 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gzk4\" (UniqueName: \"kubernetes.io/projected/94105113-4ed4-4985-86bf-911aa03ba7fd-kube-api-access-8gzk4\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:59:34.933203 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.933164 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc"] Sep 11 14:59:34.940927 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.940897 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-362f7a-6b475b4b95-xwprc"] Sep 11 14:59:34.953775 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.953750 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj"] Sep 11 14:59:34.956211 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:34.956190 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-362f7a-849fb7b974-tplmj"] Sep 11 14:59:35.331614 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:35.331533 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e861242-c72a-48df-a49f-654e107cbd3c" path="/var/lib/kubelet/pods/8e861242-c72a-48df-a49f-654e107cbd3c/volumes" Sep 11 14:59:35.331951 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:35.331936 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94105113-4ed4-4985-86bf-911aa03ba7fd" path="/var/lib/kubelet/pods/94105113-4ed4-4985-86bf-911aa03ba7fd/volumes" Sep 11 14:59:35.332238 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:35.332226 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a990bbc0-7844-44ad-a0d8-ac7e038846d5" path="/var/lib/kubelet/pods/a990bbc0-7844-44ad-a0d8-ac7e038846d5/volumes" Sep 11 14:59:42.205299 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:42.205264 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90d6f5-cd4mf"] Sep 11 14:59:42.205720 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:42.205691 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e861242-c72a-48df-a49f-654e107cbd3c" containerName="payload-pre-processing" Sep 11 14:59:42.205720 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:42.205720 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e861242-c72a-48df-a49f-654e107cbd3c" containerName="payload-pre-processing" Sep 11 14:59:42.205818 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:42.205731 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94105113-4ed4-4985-86bf-911aa03ba7fd" containerName="payload-processing" Sep 11 14:59:42.205818 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:42.205737 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="94105113-4ed4-4985-86bf-911aa03ba7fd" containerName="payload-processing" Sep 11 14:59:42.205818 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:42.205809 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e861242-c72a-48df-a49f-654e107cbd3c" containerName="payload-pre-processing" Sep 11 14:59:42.205818 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:42.205819 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="94105113-4ed4-4985-86bf-911aa03ba7fd" containerName="payload-processing" Sep 11 14:59:42.208943 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:42.208927 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90d6f5-cd4mf" Sep 11 14:59:42.211105 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:42.211079 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:59:42.211105 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:42.211103 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nks8w\"" Sep 11 14:59:42.216803 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:42.216499 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90d6f5-cd4mf"] Sep 11 14:59:42.287681 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:42.287655 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91093546-dd38-46d4-9e10-59ea622c19f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-90d6f5-cd4mf\" (UID: \"91093546-dd38-46d4-9e10-59ea622c19f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90d6f5-cd4mf" Sep 11 14:59:42.388469 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:42.388440 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91093546-dd38-46d4-9e10-59ea622c19f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-90d6f5-cd4mf\" (UID: \"91093546-dd38-46d4-9e10-59ea622c19f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90d6f5-cd4mf" Sep 11 14:59:42.389056 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:42.389035 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91093546-dd38-46d4-9e10-59ea622c19f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-90d6f5-cd4mf\" (UID: \"91093546-dd38-46d4-9e10-59ea622c19f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90d6f5-cd4mf" Sep 11 14:59:42.520800 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:42.520679 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90d6f5-cd4mf" Sep 11 14:59:42.793649 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:42.793099 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90d6f5-cd4mf"] Sep 11 14:59:42.796832 ip-10-0-135-39 kubenswrapper[2583]: W0911 14:59:42.796796 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91093546_dd38_46d4_9e10_59ea622c19f5.slice/crio-f1e9a54baa38b8a2d970391bafb292be95efd736a403fd648dbd4e82ad0405d0 WatchSource:0}: Error finding container f1e9a54baa38b8a2d970391bafb292be95efd736a403fd648dbd4e82ad0405d0: Status 404 returned error can't find the container with id f1e9a54baa38b8a2d970391bafb292be95efd736a403fd648dbd4e82ad0405d0 Sep 11 14:59:43.667687 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:43.667650 2583 generic.go:356] "Generic (PLEG): container finished" podID="91093546-dd38-46d4-9e10-59ea622c19f5" containerID="967a4ea78e000f6ca3b3c21fa6994bcc85cd04b965d8677cf09e83b71a22ebcb" exitCode=0 Sep 11 14:59:43.668103 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:43.667726 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90d6f5-cd4mf" event={"ID":"91093546-dd38-46d4-9e10-59ea622c19f5","Type":"ContainerDied","Data":"967a4ea78e000f6ca3b3c21fa6994bcc85cd04b965d8677cf09e83b71a22ebcb"} Sep 11 14:59:43.668103 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:43.667787 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90d6f5-cd4mf" event={"ID":"91093546-dd38-46d4-9e10-59ea622c19f5","Type":"ContainerStarted","Data":"f1e9a54baa38b8a2d970391bafb292be95efd736a403fd648dbd4e82ad0405d0"} Sep 11 14:59:44.607096 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:44.607060 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt" Sep 11 14:59:44.607836 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:44.607813 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl" Sep 11 14:59:44.808045 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:44.808019 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90d6f5-cd4mf" Sep 11 14:59:44.914995 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:44.914964 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91093546-dd38-46d4-9e10-59ea622c19f5-maas-api-service-ca\") pod \"91093546-dd38-46d4-9e10-59ea622c19f5\" (UID: \"91093546-dd38-46d4-9e10-59ea622c19f5\") " Sep 11 14:59:44.915333 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:44.915310 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91093546-dd38-46d4-9e10-59ea622c19f5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "91093546-dd38-46d4-9e10-59ea622c19f5" (UID: "91093546-dd38-46d4-9e10-59ea622c19f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:59:45.016384 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:45.016350 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91093546-dd38-46d4-9e10-59ea622c19f5-maas-api-service-ca\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 14:59:45.680616 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:45.680575 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90d6f5-cd4mf" event={"ID":"91093546-dd38-46d4-9e10-59ea622c19f5","Type":"ContainerDied","Data":"f1e9a54baa38b8a2d970391bafb292be95efd736a403fd648dbd4e82ad0405d0"} Sep 11 14:59:45.680616 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:45.680596 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90d6f5-cd4mf" Sep 11 14:59:45.680616 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:45.680623 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1e9a54baa38b8a2d970391bafb292be95efd736a403fd648dbd4e82ad0405d0" Sep 11 14:59:54.089631 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:54.089592 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90d6f5-cd4mf"] Sep 11 14:59:54.093106 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:54.093079 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-90d6f5-cd4mf"] Sep 11 14:59:55.330878 ip-10-0-135-39 kubenswrapper[2583]: I0911 14:59:55.330845 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91093546-dd38-46d4-9e10-59ea622c19f5" path="/var/lib/kubelet/pods/91093546-dd38-46d4-9e10-59ea622c19f5/volumes" Sep 11 15:00:00.148066 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.148017 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818980-7fctb"] Sep 11 15:00:00.148522 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.148459 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91093546-dd38-46d4-9e10-59ea622c19f5" containerName="revoke-keys" Sep 11 15:00:00.148522 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.148474 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="91093546-dd38-46d4-9e10-59ea622c19f5" containerName="revoke-keys" Sep 11 15:00:00.148627 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.148542 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="91093546-dd38-46d4-9e10-59ea622c19f5" containerName="revoke-keys" Sep 11 15:00:00.151928 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.151911 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e7ca56-29818980-fdkwq"] Sep 11 15:00:00.152049 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.152033 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818980-7fctb" Sep 11 15:00:00.155376 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.155153 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nks8w\"" Sep 11 15:00:00.161745 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.161723 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e7ca56-29818980-fdkwq" Sep 11 15:00:00.163636 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.163071 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c4e39b-29818980-wg887"] Sep 11 15:00:00.168552 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.168520 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818980-7fctb" Sep 11 15:00:00.175484 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.175463 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818980-7fctb"] Sep 11 15:00:00.176437 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.175503 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c4e39b-29818980-wg887"] Sep 11 15:00:00.176437 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.175573 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c4e39b-29818980-wg887" Sep 11 15:00:00.177908 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.177887 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e7ca56-29818980-fdkwq"] Sep 11 15:00:00.181120 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.181102 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c0e28e-29818980-nxpmt"] Sep 11 15:00:00.184838 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.184819 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e7ca56-29818980-fdkwq" Sep 11 15:00:00.189099 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.186935 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c0e28e-29818980-nxpmt" Sep 11 15:00:00.189099 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.188806 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c0e28e-29818980-nxpmt"] Sep 11 15:00:00.225387 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.225358 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c4e39b-29818980-wg887" Sep 11 15:00:00.238283 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.238063 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c0e28e-29818980-nxpmt" Sep 11 15:00:00.458157 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.458131 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818980-7fctb"] Sep 11 15:00:00.459502 ip-10-0-135-39 kubenswrapper[2583]: W0911 15:00:00.459477 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e8255c_0236_46e7_8bb8_c3ab6fde2d8b.slice/crio-df09084828a129fb4aa2195ceb12bd66ddc0e1fc06a754bf14632f52cfb9fd81 WatchSource:0}: Error finding container df09084828a129fb4aa2195ceb12bd66ddc0e1fc06a754bf14632f52cfb9fd81: Status 404 returned error can't find the container with id df09084828a129fb4aa2195ceb12bd66ddc0e1fc06a754bf14632f52cfb9fd81 Sep 11 15:00:00.475368 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.475344 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e7ca56-29818980-fdkwq"] Sep 11 15:00:00.476330 ip-10-0-135-39 kubenswrapper[2583]: W0911 15:00:00.476306 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b6393e1_cbbf_4a39_9580_d4b51c427560.slice/crio-faded5d11e953e85ad079db22d4e442919bdd80efcda9a043a3c814b38cab4a7 WatchSource:0}: Error finding container faded5d11e953e85ad079db22d4e442919bdd80efcda9a043a3c814b38cab4a7: Status 404 returned error can't find the container with id faded5d11e953e85ad079db22d4e442919bdd80efcda9a043a3c814b38cab4a7 Sep 11 15:00:00.538317 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.538294 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c4e39b-29818980-wg887"] Sep 11 15:00:00.539167 ip-10-0-135-39 kubenswrapper[2583]: W0911 15:00:00.539146 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd476d701_ae6b_40bd_9054_efada8ccd100.slice/crio-af6760ce8d53c37d9fd79b6cb2e5b8ccb6e3a4e2edba09835dd7fdf44e55dfe1 WatchSource:0}: Error finding container af6760ce8d53c37d9fd79b6cb2e5b8ccb6e3a4e2edba09835dd7fdf44e55dfe1: Status 404 returned error can't find the container with id af6760ce8d53c37d9fd79b6cb2e5b8ccb6e3a4e2edba09835dd7fdf44e55dfe1 Sep 11 15:00:00.552955 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.552925 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c0e28e-29818980-nxpmt"] Sep 11 15:00:00.554874 ip-10-0-135-39 kubenswrapper[2583]: W0911 15:00:00.554852 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd5471a_377b_4400_b1c4_eedb771ac1b4.slice/crio-f6321a0d8b372373657570a0e2e1d2584f10cffbc4680b9b92513e6dc130697e WatchSource:0}: Error finding container f6321a0d8b372373657570a0e2e1d2584f10cffbc4680b9b92513e6dc130697e: Status 404 returned error can't find the container with id f6321a0d8b372373657570a0e2e1d2584f10cffbc4680b9b92513e6dc130697e Sep 11 15:00:00.787625 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.787591 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e7ca56-29818980-fdkwq" event={"ID":"4b6393e1-cbbf-4a39-9580-d4b51c427560","Type":"ContainerStarted","Data":"1ed88ff43821c65b6ff93b59572385e601b160811f6972268cdc626fa7b46c09"} Sep 11 15:00:00.787779 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.787649 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e7ca56-29818980-fdkwq" event={"ID":"4b6393e1-cbbf-4a39-9580-d4b51c427560","Type":"ContainerStarted","Data":"faded5d11e953e85ad079db22d4e442919bdd80efcda9a043a3c814b38cab4a7"} Sep 11 15:00:00.788955 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.788933 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c4e39b-29818980-wg887" event={"ID":"d476d701-ae6b-40bd-9054-efada8ccd100","Type":"ContainerStarted","Data":"af6760ce8d53c37d9fd79b6cb2e5b8ccb6e3a4e2edba09835dd7fdf44e55dfe1"} Sep 11 15:00:00.790190 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.790167 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c0e28e-29818980-nxpmt" event={"ID":"6fd5471a-377b-4400-b1c4-eedb771ac1b4","Type":"ContainerStarted","Data":"f6321a0d8b372373657570a0e2e1d2584f10cffbc4680b9b92513e6dc130697e"} Sep 11 15:00:00.791617 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.791595 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818980-7fctb" event={"ID":"86e8255c-0236-46e7-8bb8-c3ab6fde2d8b","Type":"ContainerStarted","Data":"f94f3640a4d7fed418751bf9564ffe8e1be1cdb2ca217f3c16485d3bc722345c"} Sep 11 15:00:00.791721 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.791631 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818980-7fctb" event={"ID":"86e8255c-0236-46e7-8bb8-c3ab6fde2d8b","Type":"ContainerStarted","Data":"df09084828a129fb4aa2195ceb12bd66ddc0e1fc06a754bf14632f52cfb9fd81"} Sep 11 15:00:00.811479 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.811421 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e7ca56-29818980-fdkwq" podStartSLOduration=0.811398929 podStartE2EDuration="811.398929ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:00:00.804842786 +0000 UTC m=+1192.048828473" watchObservedRunningTime="2026-09-11 15:00:00.811398929 +0000 UTC m=+1192.055384589" Sep 11 15:00:00.821816 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:00.821765 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818980-7fctb" podStartSLOduration=0.821752688 podStartE2EDuration="821.752688ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:00:00.82130346 +0000 UTC m=+1192.065289130" watchObservedRunningTime="2026-09-11 15:00:00.821752688 +0000 UTC m=+1192.065738338" Sep 11 15:00:01.800802 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:01.800770 2583 generic.go:356] "Generic (PLEG): container finished" podID="d476d701-ae6b-40bd-9054-efada8ccd100" containerID="bb7a24390d0a5460b0bda3ed0a31788544fd9029ae17bfd698eb36355d2da909" exitCode=0 Sep 11 15:00:01.801232 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:01.800851 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c4e39b-29818980-wg887" event={"ID":"d476d701-ae6b-40bd-9054-efada8ccd100","Type":"ContainerDied","Data":"bb7a24390d0a5460b0bda3ed0a31788544fd9029ae17bfd698eb36355d2da909"} Sep 11 15:00:01.802581 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:01.802545 2583 generic.go:356] "Generic (PLEG): container finished" podID="6fd5471a-377b-4400-b1c4-eedb771ac1b4" containerID="7646db0c1de38dc73076aa505b84ef7053f2701070783e1b8e263afeafa57a7d" exitCode=0 Sep 11 15:00:01.802722 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:01.802616 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c0e28e-29818980-nxpmt" event={"ID":"6fd5471a-377b-4400-b1c4-eedb771ac1b4","Type":"ContainerDied","Data":"7646db0c1de38dc73076aa505b84ef7053f2701070783e1b8e263afeafa57a7d"} Sep 11 15:00:01.804189 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:01.804168 2583 generic.go:356] "Generic (PLEG): container finished" podID="86e8255c-0236-46e7-8bb8-c3ab6fde2d8b" containerID="f94f3640a4d7fed418751bf9564ffe8e1be1cdb2ca217f3c16485d3bc722345c" exitCode=0 Sep 11 15:00:01.804289 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:01.804237 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818980-7fctb" event={"ID":"86e8255c-0236-46e7-8bb8-c3ab6fde2d8b","Type":"ContainerDied","Data":"f94f3640a4d7fed418751bf9564ffe8e1be1cdb2ca217f3c16485d3bc722345c"} Sep 11 15:00:01.805638 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:01.805617 2583 generic.go:356] "Generic (PLEG): container finished" podID="4b6393e1-cbbf-4a39-9580-d4b51c427560" containerID="1ed88ff43821c65b6ff93b59572385e601b160811f6972268cdc626fa7b46c09" exitCode=0 Sep 11 15:00:01.805735 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:01.805700 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e7ca56-29818980-fdkwq" event={"ID":"4b6393e1-cbbf-4a39-9580-d4b51c427560","Type":"ContainerDied","Data":"1ed88ff43821c65b6ff93b59572385e601b160811f6972268cdc626fa7b46c09"} Sep 11 15:00:02.950145 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:02.950104 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c4e39b-29818980-wg887" Sep 11 15:00:02.987297 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:02.987264 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e7ca56-29818980-fdkwq" Sep 11 15:00:03.037127 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:03.037105 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818980-7fctb" Sep 11 15:00:03.040327 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:03.040305 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c0e28e-29818980-nxpmt" Sep 11 15:00:03.821540 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:03.821510 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c4e39b-29818980-wg887" Sep 11 15:00:03.821738 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:03.821545 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c4e39b-29818980-wg887" event={"ID":"d476d701-ae6b-40bd-9054-efada8ccd100","Type":"ContainerDied","Data":"af6760ce8d53c37d9fd79b6cb2e5b8ccb6e3a4e2edba09835dd7fdf44e55dfe1"} Sep 11 15:00:03.821738 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:03.821598 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af6760ce8d53c37d9fd79b6cb2e5b8ccb6e3a4e2edba09835dd7fdf44e55dfe1" Sep 11 15:00:03.823049 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:03.823030 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c0e28e-29818980-nxpmt" Sep 11 15:00:03.823168 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:03.823056 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c0e28e-29818980-nxpmt" event={"ID":"6fd5471a-377b-4400-b1c4-eedb771ac1b4","Type":"ContainerDied","Data":"f6321a0d8b372373657570a0e2e1d2584f10cffbc4680b9b92513e6dc130697e"} Sep 11 15:00:03.823168 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:03.823096 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6321a0d8b372373657570a0e2e1d2584f10cffbc4680b9b92513e6dc130697e" Sep 11 15:00:03.824553 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:03.824536 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818980-7fctb" Sep 11 15:00:03.824553 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:03.824545 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818980-7fctb" event={"ID":"86e8255c-0236-46e7-8bb8-c3ab6fde2d8b","Type":"ContainerDied","Data":"df09084828a129fb4aa2195ceb12bd66ddc0e1fc06a754bf14632f52cfb9fd81"} Sep 11 15:00:03.824749 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:03.824581 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df09084828a129fb4aa2195ceb12bd66ddc0e1fc06a754bf14632f52cfb9fd81" Sep 11 15:00:03.826034 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:03.826015 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e7ca56-29818980-fdkwq" Sep 11 15:00:03.826034 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:03.826025 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e7ca56-29818980-fdkwq" event={"ID":"4b6393e1-cbbf-4a39-9580-d4b51c427560","Type":"ContainerDied","Data":"faded5d11e953e85ad079db22d4e442919bdd80efcda9a043a3c814b38cab4a7"} Sep 11 15:00:03.826154 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:03.826059 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faded5d11e953e85ad079db22d4e442919bdd80efcda9a043a3c814b38cab4a7" Sep 11 15:00:09.259226 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:09.259199 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lwrt2_57d4ba26-384d-42f8-86a1-335e27cdf762/console-operator/1.log" Sep 11 15:00:09.262876 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:09.262856 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lwrt2_57d4ba26-384d-42f8-86a1-335e27cdf762/console-operator/1.log" Sep 11 15:00:09.361251 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:09.361224 2583 scope.go:117] "RemoveContainer" containerID="83326b5004e1793d554eaa7eb406fb3caa2fcc8dcca34e8ad8e0c5fce6c83364" Sep 11 15:00:37.029733 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:37.029670 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6"] Sep 11 15:00:37.030265 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:37.029933 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" podUID="3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb" containerName="istio-proxy" containerID="cri-o://a61a29e40ffc7fe6b414376c8c429f9a6757ae30a5033c2c5aacadff1e2e9281" gracePeriod=30 Sep 11 15:00:38.052721 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.052678 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5"] Sep 11 15:00:38.053303 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.053282 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fd5471a-377b-4400-b1c4-eedb771ac1b4" containerName="cleanup" Sep 11 15:00:38.053303 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.053305 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fd5471a-377b-4400-b1c4-eedb771ac1b4" containerName="cleanup" Sep 11 15:00:38.053452 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.053325 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b6393e1-cbbf-4a39-9580-d4b51c427560" containerName="cleanup" Sep 11 15:00:38.053452 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.053334 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b6393e1-cbbf-4a39-9580-d4b51c427560" containerName="cleanup" Sep 11 15:00:38.053452 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.053348 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d476d701-ae6b-40bd-9054-efada8ccd100" containerName="cleanup" Sep 11 15:00:38.053452 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.053357 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d476d701-ae6b-40bd-9054-efada8ccd100" containerName="cleanup" Sep 11 15:00:38.053452 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.053369 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86e8255c-0236-46e7-8bb8-c3ab6fde2d8b" containerName="cleanup" Sep 11 15:00:38.053452 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.053377 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="86e8255c-0236-46e7-8bb8-c3ab6fde2d8b" containerName="cleanup" Sep 11 15:00:38.053672 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.053471 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d476d701-ae6b-40bd-9054-efada8ccd100" containerName="cleanup" Sep 11 15:00:38.053672 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.053485 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fd5471a-377b-4400-b1c4-eedb771ac1b4" containerName="cleanup" Sep 11 15:00:38.053672 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.053495 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b6393e1-cbbf-4a39-9580-d4b51c427560" containerName="cleanup" Sep 11 15:00:38.053672 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.053507 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="86e8255c-0236-46e7-8bb8-c3ab6fde2d8b" containerName="cleanup" Sep 11 15:00:38.061911 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.061875 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.066500 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.066472 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-48898f-openshift-default-dockercfg-p7fk6\"" Sep 11 15:00:38.071560 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.071536 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5"] Sep 11 15:00:38.112906 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.112876 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-workload-socket\") pod \"e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.113017 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.112923 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-istio-envoy\") pod \"e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.113017 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.112954 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g78j\" (UniqueName: \"kubernetes.io/projected/f03699c7-057d-4bce-a174-a5784cfcef79-kube-api-access-7g78j\") pod \"e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.113017 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.112977 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f03699c7-057d-4bce-a174-a5784cfcef79-istio-token\") pod \"e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.113128 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.113041 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f03699c7-057d-4bce-a174-a5784cfcef79-istiod-ca-cert\") pod \"e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.113128 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.113069 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-workload-certs\") pod \"e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.113128 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.113105 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-istio-data\") pod \"e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.113242 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.113177 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-credential-socket\") pod \"e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.113242 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.113224 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f03699c7-057d-4bce-a174-a5784cfcef79-istio-podinfo\") pod \"e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.214390 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.214351 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g78j\" (UniqueName: \"kubernetes.io/projected/f03699c7-057d-4bce-a174-a5784cfcef79-kube-api-access-7g78j\") pod \"e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.214390 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.214393 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f03699c7-057d-4bce-a174-a5784cfcef79-istio-token\") pod \"e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.214639 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.214430 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f03699c7-057d-4bce-a174-a5784cfcef79-istiod-ca-cert\") pod \"e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.214639 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.214447 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-workload-certs\") pod \"e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.214639 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.214476 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-istio-data\") pod \"e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.214639 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.214529 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-credential-socket\") pod \"e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.214639 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.214574 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f03699c7-057d-4bce-a174-a5784cfcef79-istio-podinfo\") pod \"e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.214896 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.214646 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-workload-socket\") pod \"e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.214896 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.214696 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-istio-envoy\") pod \"e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.215133 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.215103 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-workload-socket\") pod \"e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.215329 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.215156 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-workload-certs\") pod \"e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.215329 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.215188 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-istio-data\") pod \"e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.215494 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.215242 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-credential-socket\") pod \"e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.215494 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.215436 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f03699c7-057d-4bce-a174-a5784cfcef79-istiod-ca-cert\") pod \"e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.217153 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.217121 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-istio-envoy\") pod \"e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.217297 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.217279 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f03699c7-057d-4bce-a174-a5784cfcef79-istio-podinfo\") pod \"e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.222020 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.221995 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f03699c7-057d-4bce-a174-a5784cfcef79-istio-token\") pod \"e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.222138 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.222122 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g78j\" (UniqueName: \"kubernetes.io/projected/f03699c7-057d-4bce-a174-a5784cfcef79-kube-api-access-7g78j\") pod \"e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.378814 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.378781 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:38.655742 ip-10-0-135-39 kubenswrapper[2583]: W0911 15:00:38.655678 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03699c7_057d_4bce_a174_a5784cfcef79.slice/crio-debb1ec839f707eabc8d37b50823d5ca4f2b563103f614e6a581a0451796456b WatchSource:0}: Error finding container debb1ec839f707eabc8d37b50823d5ca4f2b563103f614e6a581a0451796456b: Status 404 returned error can't find the container with id debb1ec839f707eabc8d37b50823d5ca4f2b563103f614e6a581a0451796456b Sep 11 15:00:38.655742 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.655700 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5"] Sep 11 15:00:38.657920 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.657887 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 15:00:38.658021 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.657950 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 15:00:38.658021 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:38.657980 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 15:00:39.053638 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:39.053543 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" event={"ID":"f03699c7-057d-4bce-a174-a5784cfcef79","Type":"ContainerStarted","Data":"856d49a793f270c8c562dcf4b8bc4a46e4143a9d5a18d6d0b246d399323eec0e"} Sep 11 15:00:39.053638 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:39.053608 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" event={"ID":"f03699c7-057d-4bce-a174-a5784cfcef79","Type":"ContainerStarted","Data":"debb1ec839f707eabc8d37b50823d5ca4f2b563103f614e6a581a0451796456b"} Sep 11 15:00:39.080248 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:39.080202 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" podStartSLOduration=1.080186697 podStartE2EDuration="1.080186697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:00:39.076751318 +0000 UTC m=+1230.320736976" watchObservedRunningTime="2026-09-11 15:00:39.080186697 +0000 UTC m=+1230.324172356" Sep 11 15:00:39.379845 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:39.379816 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:39.384996 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:39.384969 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:40.060917 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:40.060892 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:40.061820 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:40.061799 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:00:42.281653 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.281627 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 15:00:42.355029 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.354947 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istio-data\") pod \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " Sep 11 15:00:42.355029 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.355002 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istio-token\") pod \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " Sep 11 15:00:42.355233 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.355036 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istio-podinfo\") pod \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " Sep 11 15:00:42.355233 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.355066 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-workload-socket\") pod \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " Sep 11 15:00:42.355233 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.355086 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-credential-socket\") pod \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " Sep 11 15:00:42.355233 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.355171 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istio-envoy\") pod \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " Sep 11 15:00:42.355233 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.355186 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istio-data" (OuterVolumeSpecName: "istio-data") pod "3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb" (UID: "3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:00:42.355233 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.355199 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-workload-certs\") pod \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " Sep 11 15:00:42.355528 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.355254 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istiod-ca-cert\") pod \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " Sep 11 15:00:42.355528 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.355283 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8gfs\" (UniqueName: \"kubernetes.io/projected/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-kube-api-access-m8gfs\") pod \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\" (UID: \"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb\") " Sep 11 15:00:42.355528 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.355397 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb" (UID: "3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:00:42.355528 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.355455 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb" (UID: "3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:00:42.355528 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.355513 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb" (UID: "3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:00:42.355871 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.355788 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-workload-certs\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:00:42.355871 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.355810 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istio-data\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:00:42.355871 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.355823 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-workload-socket\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:00:42.355871 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.355823 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb" (UID: "3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:00:42.355871 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.355835 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-credential-socket\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:00:42.357224 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.357199 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb" (UID: "3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 15:00:42.357477 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.357461 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-kube-api-access-m8gfs" (OuterVolumeSpecName: "kube-api-access-m8gfs") pod "3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb" (UID: "3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb"). InnerVolumeSpecName "kube-api-access-m8gfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:00:42.357617 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.357590 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb" (UID: "3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:00:42.357729 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.357594 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istio-token" (OuterVolumeSpecName: "istio-token") pod "3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb" (UID: "3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:00:42.456372 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.456332 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istio-envoy\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:00:42.456372 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.456371 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istiod-ca-cert\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:00:42.456372 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.456382 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8gfs\" (UniqueName: \"kubernetes.io/projected/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-kube-api-access-m8gfs\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:00:42.456573 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.456391 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istio-token\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:00:42.456573 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:42.456400 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb-istio-podinfo\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:00:43.085348 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:43.085308 2583 generic.go:356] "Generic (PLEG): container finished" podID="3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb" containerID="a61a29e40ffc7fe6b414376c8c429f9a6757ae30a5033c2c5aacadff1e2e9281" exitCode=0 Sep 11 15:00:43.085523 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:43.085363 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" Sep 11 15:00:43.085523 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:43.085393 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" event={"ID":"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb","Type":"ContainerDied","Data":"a61a29e40ffc7fe6b414376c8c429f9a6757ae30a5033c2c5aacadff1e2e9281"} Sep 11 15:00:43.085523 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:43.085450 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6" event={"ID":"3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb","Type":"ContainerDied","Data":"7436ea1252b16f602ad7074dd04778936388925a30ad3f614d6187cf29a3734f"} Sep 11 15:00:43.085523 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:43.085475 2583 scope.go:117] "RemoveContainer" containerID="a61a29e40ffc7fe6b414376c8c429f9a6757ae30a5033c2c5aacadff1e2e9281" Sep 11 15:00:43.095841 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:43.095821 2583 scope.go:117] "RemoveContainer" containerID="357ca16ae385a49a5476eea7e77bc433134a77dafee8a784a45a3e488de30e4e" Sep 11 15:00:43.104487 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:43.104466 2583 scope.go:117] "RemoveContainer" containerID="a61a29e40ffc7fe6b414376c8c429f9a6757ae30a5033c2c5aacadff1e2e9281" Sep 11 15:00:43.104769 ip-10-0-135-39 kubenswrapper[2583]: E0911 15:00:43.104748 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61a29e40ffc7fe6b414376c8c429f9a6757ae30a5033c2c5aacadff1e2e9281\": container with ID starting with a61a29e40ffc7fe6b414376c8c429f9a6757ae30a5033c2c5aacadff1e2e9281 not found: ID does not exist" containerID="a61a29e40ffc7fe6b414376c8c429f9a6757ae30a5033c2c5aacadff1e2e9281" Sep 11 15:00:43.104859 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:43.104780 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61a29e40ffc7fe6b414376c8c429f9a6757ae30a5033c2c5aacadff1e2e9281"} err="failed to get container status \"a61a29e40ffc7fe6b414376c8c429f9a6757ae30a5033c2c5aacadff1e2e9281\": rpc error: code = NotFound desc = could not find container \"a61a29e40ffc7fe6b414376c8c429f9a6757ae30a5033c2c5aacadff1e2e9281\": container with ID starting with a61a29e40ffc7fe6b414376c8c429f9a6757ae30a5033c2c5aacadff1e2e9281 not found: ID does not exist" Sep 11 15:00:43.104859 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:43.104803 2583 scope.go:117] "RemoveContainer" containerID="357ca16ae385a49a5476eea7e77bc433134a77dafee8a784a45a3e488de30e4e" Sep 11 15:00:43.105119 ip-10-0-135-39 kubenswrapper[2583]: E0911 15:00:43.105099 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"357ca16ae385a49a5476eea7e77bc433134a77dafee8a784a45a3e488de30e4e\": container with ID starting with 357ca16ae385a49a5476eea7e77bc433134a77dafee8a784a45a3e488de30e4e not found: ID does not exist" containerID="357ca16ae385a49a5476eea7e77bc433134a77dafee8a784a45a3e488de30e4e" Sep 11 15:00:43.105177 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:43.105124 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"357ca16ae385a49a5476eea7e77bc433134a77dafee8a784a45a3e488de30e4e"} err="failed to get container status \"357ca16ae385a49a5476eea7e77bc433134a77dafee8a784a45a3e488de30e4e\": rpc error: code = NotFound desc = could not find container \"357ca16ae385a49a5476eea7e77bc433134a77dafee8a784a45a3e488de30e4e\": container with ID starting with 357ca16ae385a49a5476eea7e77bc433134a77dafee8a784a45a3e488de30e4e not found: ID does not exist" Sep 11 15:00:43.108990 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:43.108970 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6"] Sep 11 15:00:43.112974 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:43.112953 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-90d6f5-openshift-default-77b48b5f6d-r9pp6"] Sep 11 15:00:43.331472 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:43.331446 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb" path="/var/lib/kubelet/pods/3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb/volumes" Sep 11 15:00:46.020304 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:46.020275 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j"] Sep 11 15:00:46.020678 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:46.020665 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb" containerName="istio-proxy" Sep 11 15:00:46.020678 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:46.020679 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb" containerName="istio-proxy" Sep 11 15:00:46.020778 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:46.020693 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb" containerName="istio-proxy" Sep 11 15:00:46.020778 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:46.020699 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb" containerName="istio-proxy" Sep 11 15:00:46.020841 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:46.020780 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb" containerName="istio-proxy" Sep 11 15:00:46.025217 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:46.025198 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j" Sep 11 15:00:46.027344 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:46.027321 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-48898f\"" Sep 11 15:00:46.027439 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:46.027389 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-48898f-dockercfg-5vgzg\"" Sep 11 15:00:46.036921 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:46.036901 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j"] Sep 11 15:00:46.088973 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:46.088937 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74156825-ef12-40fc-9862-e783ca3dd581-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j\" (UID: \"74156825-ef12-40fc-9862-e783ca3dd581\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j" Sep 11 15:00:46.089084 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:46.088989 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhrgh\" (UniqueName: \"kubernetes.io/projected/74156825-ef12-40fc-9862-e783ca3dd581-kube-api-access-mhrgh\") pod \"payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j\" (UID: \"74156825-ef12-40fc-9862-e783ca3dd581\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j" Sep 11 15:00:46.189763 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:46.189722 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhrgh\" (UniqueName: \"kubernetes.io/projected/74156825-ef12-40fc-9862-e783ca3dd581-kube-api-access-mhrgh\") pod \"payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j\" (UID: \"74156825-ef12-40fc-9862-e783ca3dd581\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j" Sep 11 15:00:46.189934 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:46.189906 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74156825-ef12-40fc-9862-e783ca3dd581-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j\" (UID: \"74156825-ef12-40fc-9862-e783ca3dd581\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j" Sep 11 15:00:46.190599 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:46.190581 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74156825-ef12-40fc-9862-e783ca3dd581-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j\" (UID: \"74156825-ef12-40fc-9862-e783ca3dd581\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j" Sep 11 15:00:46.197696 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:46.197677 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhrgh\" (UniqueName: \"kubernetes.io/projected/74156825-ef12-40fc-9862-e783ca3dd581-kube-api-access-mhrgh\") pod \"payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j\" (UID: \"74156825-ef12-40fc-9862-e783ca3dd581\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j" Sep 11 15:00:46.337200 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:46.337103 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j" Sep 11 15:00:46.616681 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:46.616658 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j"] Sep 11 15:00:46.617889 ip-10-0-135-39 kubenswrapper[2583]: W0911 15:00:46.617861 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74156825_ef12_40fc_9862_e783ca3dd581.slice/crio-98b2bc01d16b72be372c8268881d476714aeb5be395931c0c7a8ac8dd189a369 WatchSource:0}: Error finding container 98b2bc01d16b72be372c8268881d476714aeb5be395931c0c7a8ac8dd189a369: Status 404 returned error can't find the container with id 98b2bc01d16b72be372c8268881d476714aeb5be395931c0c7a8ac8dd189a369 Sep 11 15:00:47.112972 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:47.112933 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j" event={"ID":"74156825-ef12-40fc-9862-e783ca3dd581","Type":"ContainerStarted","Data":"ef6aeab5cba947d6b872ec88e396561b4a637136e92da793ad955b82803a53b0"} Sep 11 15:00:47.113462 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:47.112993 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j" event={"ID":"74156825-ef12-40fc-9862-e783ca3dd581","Type":"ContainerStarted","Data":"98b2bc01d16b72be372c8268881d476714aeb5be395931c0c7a8ac8dd189a369"} Sep 11 15:00:47.113462 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:47.113144 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j" Sep 11 15:00:47.135189 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:47.135140 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j" podStartSLOduration=1.135127912 podStartE2EDuration="1.135127912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:00:47.131622518 +0000 UTC m=+1238.375608189" watchObservedRunningTime="2026-09-11 15:00:47.135127912 +0000 UTC m=+1238.379113597" Sep 11 15:00:58.122251 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:00:58.122220 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j" Sep 11 15:01:09.278496 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:09.278464 2583 generic.go:356] "Generic (PLEG): container finished" podID="f03699c7-057d-4bce-a174-a5784cfcef79" containerID="856d49a793f270c8c562dcf4b8bc4a46e4143a9d5a18d6d0b246d399323eec0e" exitCode=0 Sep 11 15:01:09.278913 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:09.278502 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" event={"ID":"f03699c7-057d-4bce-a174-a5784cfcef79","Type":"ContainerDied","Data":"856d49a793f270c8c562dcf4b8bc4a46e4143a9d5a18d6d0b246d399323eec0e"} Sep 11 15:01:09.278959 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:09.278929 2583 scope.go:117] "RemoveContainer" containerID="856d49a793f270c8c562dcf4b8bc4a46e4143a9d5a18d6d0b246d399323eec0e" Sep 11 15:01:09.280108 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:09.280078 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 15:01:09.280177 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:09.280145 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 15:01:09.280209 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:09.280175 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 11 15:01:09.393865 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:09.393833 2583 scope.go:117] "RemoveContainer" containerID="30c16b4771b9369ff492ae512b41395552c714a9d51ae70d52ecad84b3a8c6be" Sep 11 15:01:09.405085 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:09.405065 2583 scope.go:117] "RemoveContainer" containerID="3a1b83748393cb85b8bd5928734e8112ff926851c7a24a6f788b1b9e541ded23" Sep 11 15:01:10.061492 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:10.061456 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:01:10.286620 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:10.286584 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" event={"ID":"f03699c7-057d-4bce-a174-a5784cfcef79","Type":"ContainerStarted","Data":"0789048a0f6391be4211a9930756fa9badab9aaf8d33af4db9c18b455f490041"} Sep 11 15:01:10.379919 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:10.379885 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:01:10.380885 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:10.380861 2583 patch_prober.go:28] interesting pod/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 11 15:01:10.380998 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:10.380917 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" podUID="f03699c7-057d-4bce-a174-a5784cfcef79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 11 15:01:11.379899 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:11.379861 2583 patch_prober.go:28] interesting pod/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 11 15:01:11.380354 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:11.379916 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" podUID="f03699c7-057d-4bce-a174-a5784cfcef79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 11 15:01:12.380116 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:12.380082 2583 patch_prober.go:28] interesting pod/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 11 15:01:12.380514 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:12.380133 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" podUID="f03699c7-057d-4bce-a174-a5784cfcef79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 11 15:01:13.379646 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:13.379596 2583 patch_prober.go:28] interesting pod/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 11 15:01:13.379857 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:13.379665 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" podUID="f03699c7-057d-4bce-a174-a5784cfcef79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 11 15:01:14.379263 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:14.379231 2583 patch_prober.go:28] interesting pod/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 11 15:01:14.379643 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:14.379279 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" podUID="f03699c7-057d-4bce-a174-a5784cfcef79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 11 15:01:15.379874 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:15.379828 2583 patch_prober.go:28] interesting pod/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 11 15:01:15.380231 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:15.379900 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" podUID="f03699c7-057d-4bce-a174-a5784cfcef79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 11 15:01:16.383800 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:16.383772 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:01:16.384251 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:16.384237 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:01:16.384845 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:16.384825 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:01:21.236357 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:21.236324 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5"] Sep 11 15:01:21.236844 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:21.236830 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ed8a63f-ea6d-4c4d-bae3-03a2c4004cfb" containerName="istio-proxy" Sep 11 15:01:21.239751 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:21.239733 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5" Sep 11 15:01:21.242614 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:21.242584 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-928943-dockercfg-tmrqf\"" Sep 11 15:01:21.242800 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:21.242782 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-928943\"" Sep 11 15:01:21.250861 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:21.250829 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5"] Sep 11 15:01:21.253665 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:21.253645 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6"] Sep 11 15:01:21.255874 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:21.255854 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6" Sep 11 15:01:21.271394 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:21.271372 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6"] Sep 11 15:01:21.331882 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:21.331860 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgklp\" (UniqueName: \"kubernetes.io/projected/08a86b3b-ce50-4d76-8344-af31d35d178f-kube-api-access-lgklp\") pod \"payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5\" (UID: \"08a86b3b-ce50-4d76-8344-af31d35d178f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5" Sep 11 15:01:21.332011 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:21.331905 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08a86b3b-ce50-4d76-8344-af31d35d178f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5\" (UID: \"08a86b3b-ce50-4d76-8344-af31d35d178f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5" Sep 11 15:01:21.332011 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:21.331930 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fznbq\" (UniqueName: \"kubernetes.io/projected/d613d4c4-e1cc-428c-badd-4fb01bee8c54-kube-api-access-fznbq\") pod \"payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6\" (UID: \"d613d4c4-e1cc-428c-badd-4fb01bee8c54\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6" Sep 11 15:01:21.332011 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:21.331988 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d613d4c4-e1cc-428c-badd-4fb01bee8c54-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6\" (UID: \"d613d4c4-e1cc-428c-badd-4fb01bee8c54\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6" Sep 11 15:01:21.433000 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:21.432963 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08a86b3b-ce50-4d76-8344-af31d35d178f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5\" (UID: \"08a86b3b-ce50-4d76-8344-af31d35d178f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5" Sep 11 15:01:21.433173 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:21.433009 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fznbq\" (UniqueName: \"kubernetes.io/projected/d613d4c4-e1cc-428c-badd-4fb01bee8c54-kube-api-access-fznbq\") pod \"payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6\" (UID: \"d613d4c4-e1cc-428c-badd-4fb01bee8c54\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6" Sep 11 15:01:21.433173 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:21.433036 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d613d4c4-e1cc-428c-badd-4fb01bee8c54-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6\" (UID: \"d613d4c4-e1cc-428c-badd-4fb01bee8c54\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6" Sep 11 15:01:21.433173 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:21.433117 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgklp\" (UniqueName: \"kubernetes.io/projected/08a86b3b-ce50-4d76-8344-af31d35d178f-kube-api-access-lgklp\") pod \"payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5\" (UID: \"08a86b3b-ce50-4d76-8344-af31d35d178f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5" Sep 11 15:01:21.433922 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:21.433896 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08a86b3b-ce50-4d76-8344-af31d35d178f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5\" (UID: \"08a86b3b-ce50-4d76-8344-af31d35d178f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5" Sep 11 15:01:21.433969 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:21.433921 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d613d4c4-e1cc-428c-badd-4fb01bee8c54-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6\" (UID: \"d613d4c4-e1cc-428c-badd-4fb01bee8c54\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6" Sep 11 15:01:21.441034 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:21.441006 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fznbq\" (UniqueName: \"kubernetes.io/projected/d613d4c4-e1cc-428c-badd-4fb01bee8c54-kube-api-access-fznbq\") pod \"payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6\" (UID: \"d613d4c4-e1cc-428c-badd-4fb01bee8c54\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6" Sep 11 15:01:21.441402 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:21.441384 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgklp\" (UniqueName: \"kubernetes.io/projected/08a86b3b-ce50-4d76-8344-af31d35d178f-kube-api-access-lgklp\") pod \"payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5\" (UID: \"08a86b3b-ce50-4d76-8344-af31d35d178f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5" Sep 11 15:01:21.550718 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:21.550633 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5" Sep 11 15:01:21.566090 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:21.566061 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6" Sep 11 15:01:21.828489 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:21.828461 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5"] Sep 11 15:01:21.831169 ip-10-0-135-39 kubenswrapper[2583]: W0911 15:01:21.831135 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a86b3b_ce50_4d76_8344_af31d35d178f.slice/crio-f894833bad26605cde30f28436b2a9f151fb1dabbe15bfbcee3024b59d908226 WatchSource:0}: Error finding container f894833bad26605cde30f28436b2a9f151fb1dabbe15bfbcee3024b59d908226: Status 404 returned error can't find the container with id f894833bad26605cde30f28436b2a9f151fb1dabbe15bfbcee3024b59d908226 Sep 11 15:01:21.845797 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:21.845764 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6"] Sep 11 15:01:21.846819 ip-10-0-135-39 kubenswrapper[2583]: W0911 15:01:21.846790 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd613d4c4_e1cc_428c_badd_4fb01bee8c54.slice/crio-7848f10e07a0595b124ef5e11db3c653e9b16f2d1fe01f98a8393a111d1cd3fa WatchSource:0}: Error finding container 7848f10e07a0595b124ef5e11db3c653e9b16f2d1fe01f98a8393a111d1cd3fa: Status 404 returned error can't find the container with id 7848f10e07a0595b124ef5e11db3c653e9b16f2d1fe01f98a8393a111d1cd3fa Sep 11 15:01:22.368254 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:22.368216 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6" event={"ID":"d613d4c4-e1cc-428c-badd-4fb01bee8c54","Type":"ContainerStarted","Data":"4a769ae15d45c2212e57a5dd59aa9688fd95170253a65d75f01e96ab031bd90a"} Sep 11 15:01:22.368734 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:22.368279 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6" Sep 11 15:01:22.368734 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:22.368303 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6" event={"ID":"d613d4c4-e1cc-428c-badd-4fb01bee8c54","Type":"ContainerStarted","Data":"7848f10e07a0595b124ef5e11db3c653e9b16f2d1fe01f98a8393a111d1cd3fa"} Sep 11 15:01:22.370043 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:22.370015 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5" event={"ID":"08a86b3b-ce50-4d76-8344-af31d35d178f","Type":"ContainerStarted","Data":"6888b5b36eeda00d5a15f4b80307adb3f158c8bf5dcaeff12088e236f378cce0"} Sep 11 15:01:22.370183 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:22.370059 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5" event={"ID":"08a86b3b-ce50-4d76-8344-af31d35d178f","Type":"ContainerStarted","Data":"f894833bad26605cde30f28436b2a9f151fb1dabbe15bfbcee3024b59d908226"} Sep 11 15:01:22.370183 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:22.370112 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5" Sep 11 15:01:22.384027 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:22.383976 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6" podStartSLOduration=1.3839606340000001 podStartE2EDuration="1.383960634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:01:22.383616337 +0000 UTC m=+1273.627601998" watchObservedRunningTime="2026-09-11 15:01:22.383960634 +0000 UTC m=+1273.627946294" Sep 11 15:01:22.412467 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:22.412413 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5" podStartSLOduration=1.412395362 podStartE2EDuration="1.412395362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:01:22.400633104 +0000 UTC m=+1273.644618764" watchObservedRunningTime="2026-09-11 15:01:22.412395362 +0000 UTC m=+1273.656381023" Sep 11 15:01:33.378376 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:33.378345 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6" Sep 11 15:01:33.378890 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:33.378593 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5" Sep 11 15:01:44.157299 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:44.157263 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns"] Sep 11 15:01:59.176862 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.176821 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" podUID="44b1ed25-fcc4-4eea-9d4b-725e66db1b61" containerName="main" containerID="cri-o://da2d844be0aa2962b037ca05c25e5097b5d2ca180a061a2b8102e8de84f7ea41" gracePeriod=45 Sep 11 15:01:59.417999 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.417972 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" Sep 11 15:01:59.503374 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.503280 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-home\") pod \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\" (UID: \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\") " Sep 11 15:01:59.503374 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.503346 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-tls-certs\") pod \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\" (UID: \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\") " Sep 11 15:01:59.503617 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.503405 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bggz7\" (UniqueName: \"kubernetes.io/projected/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-kube-api-access-bggz7\") pod \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\" (UID: \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\") " Sep 11 15:01:59.503617 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.503434 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-model-cache\") pod \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\" (UID: \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\") " Sep 11 15:01:59.503617 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.503459 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-dshm\") pod \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\" (UID: \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\") " Sep 11 15:01:59.503617 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.503513 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-tmp-dir\") pod \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\" (UID: \"44b1ed25-fcc4-4eea-9d4b-725e66db1b61\") " Sep 11 15:01:59.503617 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.503579 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-home" (OuterVolumeSpecName: "home") pod "44b1ed25-fcc4-4eea-9d4b-725e66db1b61" (UID: "44b1ed25-fcc4-4eea-9d4b-725e66db1b61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:01:59.503925 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.503728 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-model-cache" (OuterVolumeSpecName: "model-cache") pod "44b1ed25-fcc4-4eea-9d4b-725e66db1b61" (UID: "44b1ed25-fcc4-4eea-9d4b-725e66db1b61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:01:59.503925 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.503828 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "44b1ed25-fcc4-4eea-9d4b-725e66db1b61" (UID: "44b1ed25-fcc4-4eea-9d4b-725e66db1b61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:01:59.503925 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.503884 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-home\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:01:59.503925 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.503897 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-model-cache\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:01:59.505817 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.505787 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-kube-api-access-bggz7" (OuterVolumeSpecName: "kube-api-access-bggz7") pod "44b1ed25-fcc4-4eea-9d4b-725e66db1b61" (UID: "44b1ed25-fcc4-4eea-9d4b-725e66db1b61"). InnerVolumeSpecName "kube-api-access-bggz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:01:59.505920 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.505814 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44b1ed25-fcc4-4eea-9d4b-725e66db1b61" (UID: "44b1ed25-fcc4-4eea-9d4b-725e66db1b61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:01:59.505920 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.505824 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-dshm" (OuterVolumeSpecName: "dshm") pod "44b1ed25-fcc4-4eea-9d4b-725e66db1b61" (UID: "44b1ed25-fcc4-4eea-9d4b-725e66db1b61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:01:59.604503 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.604470 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-tmp-dir\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:01:59.604503 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.604496 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-tls-certs\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:01:59.604503 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.604507 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bggz7\" (UniqueName: \"kubernetes.io/projected/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-kube-api-access-bggz7\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:01:59.604845 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.604519 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b1ed25-fcc4-4eea-9d4b-725e66db1b61-dshm\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:01:59.614187 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.614158 2583 generic.go:356] "Generic (PLEG): container finished" podID="44b1ed25-fcc4-4eea-9d4b-725e66db1b61" containerID="da2d844be0aa2962b037ca05c25e5097b5d2ca180a061a2b8102e8de84f7ea41" exitCode=0 Sep 11 15:01:59.614341 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.614195 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" event={"ID":"44b1ed25-fcc4-4eea-9d4b-725e66db1b61","Type":"ContainerDied","Data":"da2d844be0aa2962b037ca05c25e5097b5d2ca180a061a2b8102e8de84f7ea41"} Sep 11 15:01:59.614341 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.614203 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" Sep 11 15:01:59.614341 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.614222 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns" event={"ID":"44b1ed25-fcc4-4eea-9d4b-725e66db1b61","Type":"ContainerDied","Data":"61bcf12606d6e5b3590aa327c9d11f9e9885af1150ad5e16c2a76500b48659bb"} Sep 11 15:01:59.614341 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.614247 2583 scope.go:117] "RemoveContainer" containerID="da2d844be0aa2962b037ca05c25e5097b5d2ca180a061a2b8102e8de84f7ea41" Sep 11 15:01:59.624365 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.624345 2583 scope.go:117] "RemoveContainer" containerID="da2d844be0aa2962b037ca05c25e5097b5d2ca180a061a2b8102e8de84f7ea41" Sep 11 15:01:59.624656 ip-10-0-135-39 kubenswrapper[2583]: E0911 15:01:59.624631 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da2d844be0aa2962b037ca05c25e5097b5d2ca180a061a2b8102e8de84f7ea41\": container with ID starting with da2d844be0aa2962b037ca05c25e5097b5d2ca180a061a2b8102e8de84f7ea41 not found: ID does not exist" containerID="da2d844be0aa2962b037ca05c25e5097b5d2ca180a061a2b8102e8de84f7ea41" Sep 11 15:01:59.624761 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.624659 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da2d844be0aa2962b037ca05c25e5097b5d2ca180a061a2b8102e8de84f7ea41"} err="failed to get container status \"da2d844be0aa2962b037ca05c25e5097b5d2ca180a061a2b8102e8de84f7ea41\": rpc error: code = NotFound desc = could not find container \"da2d844be0aa2962b037ca05c25e5097b5d2ca180a061a2b8102e8de84f7ea41\": container with ID starting with da2d844be0aa2962b037ca05c25e5097b5d2ca180a061a2b8102e8de84f7ea41 not found: ID does not exist" Sep 11 15:01:59.636035 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.636009 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns"] Sep 11 15:01:59.640167 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:01:59.640142 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-837632/e2e-conflict-a-372348b4-kserve-858f9f4cb4-jtqns"] Sep 11 15:02:01.330963 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:01.330929 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44b1ed25-fcc4-4eea-9d4b-725e66db1b61" path="/var/lib/kubelet/pods/44b1ed25-fcc4-4eea-9d4b-725e66db1b61/volumes" Sep 11 15:02:09.440249 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:09.440217 2583 scope.go:117] "RemoveContainer" containerID="984cccf4317b496f55ea5c1a4e6bc902b7ebbded4710233eb0174f5e0ea19a42" Sep 11 15:02:09.450026 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:09.450001 2583 scope.go:117] "RemoveContainer" containerID="f93c5df3bd430ec9124d6b0f3953065f59eb56f59bcc6fbfd4e725cec4f8bdf1" Sep 11 15:02:15.978553 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:15.978519 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7"] Sep 11 15:02:15.979032 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:15.978979 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44b1ed25-fcc4-4eea-9d4b-725e66db1b61" containerName="main" Sep 11 15:02:15.979032 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:15.978991 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="44b1ed25-fcc4-4eea-9d4b-725e66db1b61" containerName="main" Sep 11 15:02:15.979109 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:15.979058 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="44b1ed25-fcc4-4eea-9d4b-725e66db1b61" containerName="main" Sep 11 15:02:15.981607 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:15.981591 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" Sep 11 15:02:15.983994 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:15.983975 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-837632\"/\"distinct-w4-837632-kserve-self-signed-certs\"" Sep 11 15:02:15.993280 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:15.993255 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7"] Sep 11 15:02:16.069257 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:16.069220 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbd638a4-11fa-4a78-ac08-351a0e415f27-home\") pod \"distinct-w4-837632-kserve-865dc95ddb-wmht7\" (UID: \"dbd638a4-11fa-4a78-ac08-351a0e415f27\") " pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" Sep 11 15:02:16.069443 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:16.069322 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbd638a4-11fa-4a78-ac08-351a0e415f27-tmp-dir\") pod \"distinct-w4-837632-kserve-865dc95ddb-wmht7\" (UID: \"dbd638a4-11fa-4a78-ac08-351a0e415f27\") " pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" Sep 11 15:02:16.069443 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:16.069362 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbd638a4-11fa-4a78-ac08-351a0e415f27-tls-certs\") pod \"distinct-w4-837632-kserve-865dc95ddb-wmht7\" (UID: \"dbd638a4-11fa-4a78-ac08-351a0e415f27\") " pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" Sep 11 15:02:16.069443 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:16.069392 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbd638a4-11fa-4a78-ac08-351a0e415f27-model-cache\") pod \"distinct-w4-837632-kserve-865dc95ddb-wmht7\" (UID: \"dbd638a4-11fa-4a78-ac08-351a0e415f27\") " pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" Sep 11 15:02:16.069443 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:16.069418 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7v7r\" (UniqueName: \"kubernetes.io/projected/dbd638a4-11fa-4a78-ac08-351a0e415f27-kube-api-access-z7v7r\") pod \"distinct-w4-837632-kserve-865dc95ddb-wmht7\" (UID: \"dbd638a4-11fa-4a78-ac08-351a0e415f27\") " pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" Sep 11 15:02:16.069697 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:16.069449 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbd638a4-11fa-4a78-ac08-351a0e415f27-dshm\") pod \"distinct-w4-837632-kserve-865dc95ddb-wmht7\" (UID: \"dbd638a4-11fa-4a78-ac08-351a0e415f27\") " pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" Sep 11 15:02:16.170722 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:16.170664 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbd638a4-11fa-4a78-ac08-351a0e415f27-home\") pod \"distinct-w4-837632-kserve-865dc95ddb-wmht7\" (UID: \"dbd638a4-11fa-4a78-ac08-351a0e415f27\") " pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" Sep 11 15:02:16.171054 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:16.171031 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbd638a4-11fa-4a78-ac08-351a0e415f27-tmp-dir\") pod \"distinct-w4-837632-kserve-865dc95ddb-wmht7\" (UID: \"dbd638a4-11fa-4a78-ac08-351a0e415f27\") " pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" Sep 11 15:02:16.171202 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:16.171188 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbd638a4-11fa-4a78-ac08-351a0e415f27-tls-certs\") pod \"distinct-w4-837632-kserve-865dc95ddb-wmht7\" (UID: \"dbd638a4-11fa-4a78-ac08-351a0e415f27\") " pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" Sep 11 15:02:16.171853 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:16.171107 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbd638a4-11fa-4a78-ac08-351a0e415f27-home\") pod \"distinct-w4-837632-kserve-865dc95ddb-wmht7\" (UID: \"dbd638a4-11fa-4a78-ac08-351a0e415f27\") " pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" Sep 11 15:02:16.171853 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:16.171335 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbd638a4-11fa-4a78-ac08-351a0e415f27-tmp-dir\") pod \"distinct-w4-837632-kserve-865dc95ddb-wmht7\" (UID: \"dbd638a4-11fa-4a78-ac08-351a0e415f27\") " pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" Sep 11 15:02:16.172394 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:16.172353 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbd638a4-11fa-4a78-ac08-351a0e415f27-model-cache\") pod \"distinct-w4-837632-kserve-865dc95ddb-wmht7\" (UID: \"dbd638a4-11fa-4a78-ac08-351a0e415f27\") " pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" Sep 11 15:02:16.173351 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:16.171780 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbd638a4-11fa-4a78-ac08-351a0e415f27-model-cache\") pod \"distinct-w4-837632-kserve-865dc95ddb-wmht7\" (UID: \"dbd638a4-11fa-4a78-ac08-351a0e415f27\") " pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" Sep 11 15:02:16.173351 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:16.172590 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7v7r\" (UniqueName: \"kubernetes.io/projected/dbd638a4-11fa-4a78-ac08-351a0e415f27-kube-api-access-z7v7r\") pod \"distinct-w4-837632-kserve-865dc95ddb-wmht7\" (UID: \"dbd638a4-11fa-4a78-ac08-351a0e415f27\") " pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" Sep 11 15:02:16.173351 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:16.172633 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbd638a4-11fa-4a78-ac08-351a0e415f27-dshm\") pod \"distinct-w4-837632-kserve-865dc95ddb-wmht7\" (UID: \"dbd638a4-11fa-4a78-ac08-351a0e415f27\") " pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" Sep 11 15:02:16.176413 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:16.176390 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbd638a4-11fa-4a78-ac08-351a0e415f27-tls-certs\") pod \"distinct-w4-837632-kserve-865dc95ddb-wmht7\" (UID: \"dbd638a4-11fa-4a78-ac08-351a0e415f27\") " pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" Sep 11 15:02:16.176810 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:16.176773 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbd638a4-11fa-4a78-ac08-351a0e415f27-dshm\") pod \"distinct-w4-837632-kserve-865dc95ddb-wmht7\" (UID: \"dbd638a4-11fa-4a78-ac08-351a0e415f27\") " pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" Sep 11 15:02:16.181850 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:16.181825 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7v7r\" (UniqueName: \"kubernetes.io/projected/dbd638a4-11fa-4a78-ac08-351a0e415f27-kube-api-access-z7v7r\") pod \"distinct-w4-837632-kserve-865dc95ddb-wmht7\" (UID: \"dbd638a4-11fa-4a78-ac08-351a0e415f27\") " pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" Sep 11 15:02:16.291184 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:16.291100 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" Sep 11 15:02:16.568651 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:16.568625 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7"] Sep 11 15:02:16.570350 ip-10-0-135-39 kubenswrapper[2583]: W0911 15:02:16.570323 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbd638a4_11fa_4a78_ac08_351a0e415f27.slice/crio-ec3648d9c62eed1ff108da7f5012b66f071975085128ea8d6d91d6ad1549ff38 WatchSource:0}: Error finding container ec3648d9c62eed1ff108da7f5012b66f071975085128ea8d6d91d6ad1549ff38: Status 404 returned error can't find the container with id ec3648d9c62eed1ff108da7f5012b66f071975085128ea8d6d91d6ad1549ff38 Sep 11 15:02:16.735919 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:16.735878 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" event={"ID":"dbd638a4-11fa-4a78-ac08-351a0e415f27","Type":"ContainerStarted","Data":"47f96d0a9f5790586bf4a2d82cd9d67821b33aa3cb29948af726121954bc5837"} Sep 11 15:02:16.736114 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:16.735925 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" event={"ID":"dbd638a4-11fa-4a78-ac08-351a0e415f27","Type":"ContainerStarted","Data":"ec3648d9c62eed1ff108da7f5012b66f071975085128ea8d6d91d6ad1549ff38"} Sep 11 15:02:16.754870 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:16.754824 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" podStartSLOduration=1.754808717 podStartE2EDuration="1.754808717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:02:16.753681847 +0000 UTC m=+1327.997667504" watchObservedRunningTime="2026-09-11 15:02:16.754808717 +0000 UTC m=+1327.998794440" Sep 11 15:02:25.396025 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:25.395982 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a843ae03-7nqbr"] Sep 11 15:02:25.399019 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:25.399000 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a843ae03-7nqbr" Sep 11 15:02:25.401785 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:25.401758 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:02:25.401915 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:25.401809 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nks8w\"" Sep 11 15:02:25.407942 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:25.407484 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a843ae03-7nqbr"] Sep 11 15:02:25.460590 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:25.460555 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96abfe04-6187-4a86-b5a7-3d84daa1bac5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a843ae03-7nqbr\" (UID: \"96abfe04-6187-4a86-b5a7-3d84daa1bac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a843ae03-7nqbr" Sep 11 15:02:25.561664 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:25.561631 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96abfe04-6187-4a86-b5a7-3d84daa1bac5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a843ae03-7nqbr\" (UID: \"96abfe04-6187-4a86-b5a7-3d84daa1bac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a843ae03-7nqbr" Sep 11 15:02:25.562225 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:25.562207 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96abfe04-6187-4a86-b5a7-3d84daa1bac5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a843ae03-7nqbr\" (UID: \"96abfe04-6187-4a86-b5a7-3d84daa1bac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a843ae03-7nqbr" Sep 11 15:02:25.712796 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:25.712685 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a843ae03-7nqbr" Sep 11 15:02:25.986449 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:25.986421 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a843ae03-7nqbr"] Sep 11 15:02:25.988190 ip-10-0-135-39 kubenswrapper[2583]: W0911 15:02:25.988161 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96abfe04_6187_4a86_b5a7_3d84daa1bac5.slice/crio-416fe41871dccca4f7684ee7bb4dfdfac2b6b3c3c1cc5a22b4fdf6373e971d6a WatchSource:0}: Error finding container 416fe41871dccca4f7684ee7bb4dfdfac2b6b3c3c1cc5a22b4fdf6373e971d6a: Status 404 returned error can't find the container with id 416fe41871dccca4f7684ee7bb4dfdfac2b6b3c3c1cc5a22b4fdf6373e971d6a Sep 11 15:02:26.291245 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:26.291206 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" Sep 11 15:02:26.291245 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:26.291254 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" Sep 11 15:02:26.304316 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:26.304284 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" Sep 11 15:02:26.803244 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:26.803209 2583 generic.go:356] "Generic (PLEG): container finished" podID="96abfe04-6187-4a86-b5a7-3d84daa1bac5" containerID="cd2e2b209f5f9e732ad4ff9442e7febcaa4d4692194dc752a81d23f73ee0f2df" exitCode=0 Sep 11 15:02:26.803639 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:26.803302 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a843ae03-7nqbr" event={"ID":"96abfe04-6187-4a86-b5a7-3d84daa1bac5","Type":"ContainerDied","Data":"cd2e2b209f5f9e732ad4ff9442e7febcaa4d4692194dc752a81d23f73ee0f2df"} Sep 11 15:02:26.803639 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:26.803364 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a843ae03-7nqbr" event={"ID":"96abfe04-6187-4a86-b5a7-3d84daa1bac5","Type":"ContainerStarted","Data":"416fe41871dccca4f7684ee7bb4dfdfac2b6b3c3c1cc5a22b4fdf6373e971d6a"} Sep 11 15:02:26.814816 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:26.814791 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" Sep 11 15:02:27.944457 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:27.944431 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a843ae03-7nqbr" Sep 11 15:02:27.984531 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:27.984497 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96abfe04-6187-4a86-b5a7-3d84daa1bac5-maas-api-service-ca\") pod \"96abfe04-6187-4a86-b5a7-3d84daa1bac5\" (UID: \"96abfe04-6187-4a86-b5a7-3d84daa1bac5\") " Sep 11 15:02:27.984863 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:27.984835 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96abfe04-6187-4a86-b5a7-3d84daa1bac5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "96abfe04-6187-4a86-b5a7-3d84daa1bac5" (UID: "96abfe04-6187-4a86-b5a7-3d84daa1bac5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:02:28.085828 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:28.085730 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96abfe04-6187-4a86-b5a7-3d84daa1bac5-maas-api-service-ca\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:02:28.817009 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:28.816979 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a843ae03-7nqbr" Sep 11 15:02:28.817198 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:28.816978 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a843ae03-7nqbr" event={"ID":"96abfe04-6187-4a86-b5a7-3d84daa1bac5","Type":"ContainerDied","Data":"416fe41871dccca4f7684ee7bb4dfdfac2b6b3c3c1cc5a22b4fdf6373e971d6a"} Sep 11 15:02:28.817198 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:28.817105 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="416fe41871dccca4f7684ee7bb4dfdfac2b6b3c3c1cc5a22b4fdf6373e971d6a" Sep 11 15:02:41.253321 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:41.253203 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a843ae03-7nqbr"] Sep 11 15:02:41.256919 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:41.256895 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a843ae03-7nqbr"] Sep 11 15:02:41.330619 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:02:41.330589 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96abfe04-6187-4a86-b5a7-3d84daa1bac5" path="/var/lib/kubelet/pods/96abfe04-6187-4a86-b5a7-3d84daa1bac5/volumes" Sep 11 15:05:05.205435 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:05.205369 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh"] Sep 11 15:05:05.206181 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:05.206155 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96abfe04-6187-4a86-b5a7-3d84daa1bac5" containerName="revoke-keys" Sep 11 15:05:05.206371 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:05.206357 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="96abfe04-6187-4a86-b5a7-3d84daa1bac5" containerName="revoke-keys" Sep 11 15:05:05.206620 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:05.206603 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="96abfe04-6187-4a86-b5a7-3d84daa1bac5" containerName="revoke-keys" Sep 11 15:05:05.210830 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:05.210800 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh" Sep 11 15:05:05.213433 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:05.213407 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-54b7f9\"" Sep 11 15:05:05.213571 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:05.213407 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-54b7f9-dockercfg-wwqlg\"" Sep 11 15:05:05.219294 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:05.219270 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh"] Sep 11 15:05:05.387553 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:05.387515 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0b07e34-b488-460f-abf9-45dd0c2e0387-plugins-config-volume\") pod \"payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh\" (UID: \"b0b07e34-b488-460f-abf9-45dd0c2e0387\") " pod="openshift-ingress/payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh" Sep 11 15:05:05.387553 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:05.387558 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b85h\" (UniqueName: \"kubernetes.io/projected/b0b07e34-b488-460f-abf9-45dd0c2e0387-kube-api-access-7b85h\") pod \"payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh\" (UID: \"b0b07e34-b488-460f-abf9-45dd0c2e0387\") " pod="openshift-ingress/payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh" Sep 11 15:05:05.488574 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:05.488480 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0b07e34-b488-460f-abf9-45dd0c2e0387-plugins-config-volume\") pod \"payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh\" (UID: \"b0b07e34-b488-460f-abf9-45dd0c2e0387\") " pod="openshift-ingress/payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh" Sep 11 15:05:05.488574 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:05.488529 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b85h\" (UniqueName: \"kubernetes.io/projected/b0b07e34-b488-460f-abf9-45dd0c2e0387-kube-api-access-7b85h\") pod \"payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh\" (UID: \"b0b07e34-b488-460f-abf9-45dd0c2e0387\") " pod="openshift-ingress/payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh" Sep 11 15:05:05.489291 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:05.489269 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0b07e34-b488-460f-abf9-45dd0c2e0387-plugins-config-volume\") pod \"payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh\" (UID: \"b0b07e34-b488-460f-abf9-45dd0c2e0387\") " pod="openshift-ingress/payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh" Sep 11 15:05:05.496652 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:05.496633 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b85h\" (UniqueName: \"kubernetes.io/projected/b0b07e34-b488-460f-abf9-45dd0c2e0387-kube-api-access-7b85h\") pod \"payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh\" (UID: \"b0b07e34-b488-460f-abf9-45dd0c2e0387\") " pod="openshift-ingress/payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh" Sep 11 15:05:05.525532 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:05.525496 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh" Sep 11 15:05:05.801015 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:05.800977 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh"] Sep 11 15:05:05.803037 ip-10-0-135-39 kubenswrapper[2583]: W0911 15:05:05.803010 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b07e34_b488_460f_abf9_45dd0c2e0387.slice/crio-96d52e65b6883845297ff7a94db200ff7c84952bfccd08dec0b965aa56d299ed WatchSource:0}: Error finding container 96d52e65b6883845297ff7a94db200ff7c84952bfccd08dec0b965aa56d299ed: Status 404 returned error can't find the container with id 96d52e65b6883845297ff7a94db200ff7c84952bfccd08dec0b965aa56d299ed Sep 11 15:05:05.837291 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:05.837245 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh" event={"ID":"b0b07e34-b488-460f-abf9-45dd0c2e0387","Type":"ContainerStarted","Data":"96d52e65b6883845297ff7a94db200ff7c84952bfccd08dec0b965aa56d299ed"} Sep 11 15:05:06.845033 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:06.844993 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh" event={"ID":"b0b07e34-b488-460f-abf9-45dd0c2e0387","Type":"ContainerStarted","Data":"4defc6f7f82c686e7d37b5cb9a84ac3d2287fb0aa864e725d6d2c0972a670d80"} Sep 11 15:05:06.845488 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:06.845146 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh" Sep 11 15:05:06.864145 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:06.863942 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh" podStartSLOduration=1.8639208649999999 podStartE2EDuration="1.863920865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:05:06.86039062 +0000 UTC m=+1498.104376279" watchObservedRunningTime="2026-09-11 15:05:06.863920865 +0000 UTC m=+1498.107906527" Sep 11 15:05:06.947494 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:06.947459 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e7ca56-wqz4x"] Sep 11 15:05:06.951203 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:06.951179 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e7ca56-wqz4x" Sep 11 15:05:06.953498 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:06.953476 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nks8w\"" Sep 11 15:05:06.953612 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:06.953475 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:05:06.959367 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:06.959339 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e7ca56-wqz4x"] Sep 11 15:05:06.999014 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:06.998979 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31b708e9-eb7c-47ee-b177-78c6bc3b97b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e7ca56-wqz4x\" (UID: \"31b708e9-eb7c-47ee-b177-78c6bc3b97b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e7ca56-wqz4x" Sep 11 15:05:07.099756 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:07.099643 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31b708e9-eb7c-47ee-b177-78c6bc3b97b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e7ca56-wqz4x\" (UID: \"31b708e9-eb7c-47ee-b177-78c6bc3b97b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e7ca56-wqz4x" Sep 11 15:05:07.100333 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:07.100311 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31b708e9-eb7c-47ee-b177-78c6bc3b97b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e7ca56-wqz4x\" (UID: \"31b708e9-eb7c-47ee-b177-78c6bc3b97b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e7ca56-wqz4x" Sep 11 15:05:07.263059 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:07.263022 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e7ca56-wqz4x" Sep 11 15:05:07.538643 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:07.538594 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e7ca56-wqz4x"] Sep 11 15:05:07.541024 ip-10-0-135-39 kubenswrapper[2583]: W0911 15:05:07.540993 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b708e9_eb7c_47ee_b177_78c6bc3b97b9.slice/crio-96bc7f9abc2a481e7d5534505de22623b2c8450c81c95b5fe76c54eede3b45db WatchSource:0}: Error finding container 96bc7f9abc2a481e7d5534505de22623b2c8450c81c95b5fe76c54eede3b45db: Status 404 returned error can't find the container with id 96bc7f9abc2a481e7d5534505de22623b2c8450c81c95b5fe76c54eede3b45db Sep 11 15:05:07.854139 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:07.854103 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e7ca56-wqz4x" event={"ID":"31b708e9-eb7c-47ee-b177-78c6bc3b97b9","Type":"ContainerStarted","Data":"79dc2b93868b841f4a942a2a0a6c315900feed34b2aae0693d68aa94b739a8ef"} Sep 11 15:05:07.854530 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:07.854161 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e7ca56-wqz4x" event={"ID":"31b708e9-eb7c-47ee-b177-78c6bc3b97b9","Type":"ContainerStarted","Data":"96bc7f9abc2a481e7d5534505de22623b2c8450c81c95b5fe76c54eede3b45db"} Sep 11 15:05:08.862612 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:08.862580 2583 generic.go:356] "Generic (PLEG): container finished" podID="31b708e9-eb7c-47ee-b177-78c6bc3b97b9" containerID="79dc2b93868b841f4a942a2a0a6c315900feed34b2aae0693d68aa94b739a8ef" exitCode=0 Sep 11 15:05:08.863081 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:08.862661 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e7ca56-wqz4x" event={"ID":"31b708e9-eb7c-47ee-b177-78c6bc3b97b9","Type":"ContainerDied","Data":"79dc2b93868b841f4a942a2a0a6c315900feed34b2aae0693d68aa94b739a8ef"} Sep 11 15:05:09.303402 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:09.303373 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lwrt2_57d4ba26-384d-42f8-86a1-335e27cdf762/console-operator/1.log" Sep 11 15:05:09.307209 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:09.307185 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lwrt2_57d4ba26-384d-42f8-86a1-335e27cdf762/console-operator/1.log" Sep 11 15:05:09.500265 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:09.500231 2583 scope.go:117] "RemoveContainer" containerID="44c57f6dc3086237ebe67590fd79b9dc17faf79d22207ddbe40ed577f0ad58df" Sep 11 15:05:09.512094 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:09.512066 2583 scope.go:117] "RemoveContainer" containerID="dacf3650f8837589615aeacf65c6036339c7c067392343575ff5391b5781765e" Sep 11 15:05:10.014918 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:10.014887 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e7ca56-wqz4x" Sep 11 15:05:10.039653 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:10.039618 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31b708e9-eb7c-47ee-b177-78c6bc3b97b9-maas-api-service-ca\") pod \"31b708e9-eb7c-47ee-b177-78c6bc3b97b9\" (UID: \"31b708e9-eb7c-47ee-b177-78c6bc3b97b9\") " Sep 11 15:05:10.039996 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:10.039972 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31b708e9-eb7c-47ee-b177-78c6bc3b97b9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "31b708e9-eb7c-47ee-b177-78c6bc3b97b9" (UID: "31b708e9-eb7c-47ee-b177-78c6bc3b97b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:05:10.140436 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:10.140394 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31b708e9-eb7c-47ee-b177-78c6bc3b97b9-maas-api-service-ca\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:05:10.876192 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:10.876158 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e7ca56-wqz4x" Sep 11 15:05:10.876192 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:10.876170 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e7ca56-wqz4x" event={"ID":"31b708e9-eb7c-47ee-b177-78c6bc3b97b9","Type":"ContainerDied","Data":"96bc7f9abc2a481e7d5534505de22623b2c8450c81c95b5fe76c54eede3b45db"} Sep 11 15:05:10.876423 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:10.876216 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96bc7f9abc2a481e7d5534505de22623b2c8450c81c95b5fe76c54eede3b45db" Sep 11 15:05:11.816362 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:11.816321 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7"] Sep 11 15:05:17.856622 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:17.856590 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh" Sep 11 15:05:23.130211 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:23.130170 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e7ca56-29818980-fdkwq"] Sep 11 15:05:23.132974 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:23.132951 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e7ca56-29818980-fdkwq"] Sep 11 15:05:23.169734 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:23.169684 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f"] Sep 11 15:05:23.169990 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:23.169942 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f" podUID="743de4e7-8ce3-4871-9d50-6f05b34bb27b" containerName="payload-pre-processing" containerID="cri-o://15c841ce5b7ae0889627e63f3c05ecfcbb5619cbd3c352dccbe52f6e2eca4f0c" gracePeriod=30 Sep 11 15:05:23.332452 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:23.332413 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b6393e1-cbbf-4a39-9580-d4b51c427560" path="/var/lib/kubelet/pods/4b6393e1-cbbf-4a39-9580-d4b51c427560/volumes" Sep 11 15:05:23.351990 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:23.351666 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e7ca56-wqz4x"] Sep 11 15:05:23.354695 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:23.354671 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e7ca56-wqz4x"] Sep 11 15:05:23.442575 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:23.442546 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f" Sep 11 15:05:23.581070 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:23.581033 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743de4e7-8ce3-4871-9d50-6f05b34bb27b-plugins-config-volume\") pod \"743de4e7-8ce3-4871-9d50-6f05b34bb27b\" (UID: \"743de4e7-8ce3-4871-9d50-6f05b34bb27b\") " Sep 11 15:05:23.581248 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:23.581101 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5d5ns\" (UniqueName: \"kubernetes.io/projected/743de4e7-8ce3-4871-9d50-6f05b34bb27b-kube-api-access-5d5ns\") pod \"743de4e7-8ce3-4871-9d50-6f05b34bb27b\" (UID: \"743de4e7-8ce3-4871-9d50-6f05b34bb27b\") " Sep 11 15:05:23.581419 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:23.581398 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/743de4e7-8ce3-4871-9d50-6f05b34bb27b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "743de4e7-8ce3-4871-9d50-6f05b34bb27b" (UID: "743de4e7-8ce3-4871-9d50-6f05b34bb27b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:05:23.581488 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:23.581477 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743de4e7-8ce3-4871-9d50-6f05b34bb27b-plugins-config-volume\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:05:23.583239 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:23.583213 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/743de4e7-8ce3-4871-9d50-6f05b34bb27b-kube-api-access-5d5ns" (OuterVolumeSpecName: "kube-api-access-5d5ns") pod "743de4e7-8ce3-4871-9d50-6f05b34bb27b" (UID: "743de4e7-8ce3-4871-9d50-6f05b34bb27b"). InnerVolumeSpecName "kube-api-access-5d5ns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:05:23.682136 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:23.682033 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d5ns\" (UniqueName: \"kubernetes.io/projected/743de4e7-8ce3-4871-9d50-6f05b34bb27b-kube-api-access-5d5ns\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:05:23.972957 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:23.972868 2583 generic.go:356] "Generic (PLEG): container finished" podID="743de4e7-8ce3-4871-9d50-6f05b34bb27b" containerID="15c841ce5b7ae0889627e63f3c05ecfcbb5619cbd3c352dccbe52f6e2eca4f0c" exitCode=0 Sep 11 15:05:23.972957 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:23.972915 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f" Sep 11 15:05:23.973147 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:23.972955 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f" event={"ID":"743de4e7-8ce3-4871-9d50-6f05b34bb27b","Type":"ContainerDied","Data":"15c841ce5b7ae0889627e63f3c05ecfcbb5619cbd3c352dccbe52f6e2eca4f0c"} Sep 11 15:05:23.973147 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:23.973014 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f" event={"ID":"743de4e7-8ce3-4871-9d50-6f05b34bb27b","Type":"ContainerDied","Data":"c3d7f2d8af0808d4d072ded9907f3375d927339857130d48ba2cf88d115a46f2"} Sep 11 15:05:23.973147 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:23.973043 2583 scope.go:117] "RemoveContainer" containerID="15c841ce5b7ae0889627e63f3c05ecfcbb5619cbd3c352dccbe52f6e2eca4f0c" Sep 11 15:05:23.983405 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:23.983373 2583 scope.go:117] "RemoveContainer" containerID="15c841ce5b7ae0889627e63f3c05ecfcbb5619cbd3c352dccbe52f6e2eca4f0c" Sep 11 15:05:23.983684 ip-10-0-135-39 kubenswrapper[2583]: E0911 15:05:23.983666 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15c841ce5b7ae0889627e63f3c05ecfcbb5619cbd3c352dccbe52f6e2eca4f0c\": container with ID starting with 15c841ce5b7ae0889627e63f3c05ecfcbb5619cbd3c352dccbe52f6e2eca4f0c not found: ID does not exist" containerID="15c841ce5b7ae0889627e63f3c05ecfcbb5619cbd3c352dccbe52f6e2eca4f0c" Sep 11 15:05:23.983753 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:23.983694 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c841ce5b7ae0889627e63f3c05ecfcbb5619cbd3c352dccbe52f6e2eca4f0c"} err="failed to get container status \"15c841ce5b7ae0889627e63f3c05ecfcbb5619cbd3c352dccbe52f6e2eca4f0c\": rpc error: code = NotFound desc = could not find container \"15c841ce5b7ae0889627e63f3c05ecfcbb5619cbd3c352dccbe52f6e2eca4f0c\": container with ID starting with 15c841ce5b7ae0889627e63f3c05ecfcbb5619cbd3c352dccbe52f6e2eca4f0c not found: ID does not exist" Sep 11 15:05:24.013119 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:24.013095 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f"] Sep 11 15:05:24.017968 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:24.017944 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e7ca56-69c6dcc98b-lzc2f"] Sep 11 15:05:25.331365 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:25.331334 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31b708e9-eb7c-47ee-b177-78c6bc3b97b9" path="/var/lib/kubelet/pods/31b708e9-eb7c-47ee-b177-78c6bc3b97b9/volumes" Sep 11 15:05:25.331768 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:25.331693 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="743de4e7-8ce3-4871-9d50-6f05b34bb27b" path="/var/lib/kubelet/pods/743de4e7-8ce3-4871-9d50-6f05b34bb27b/volumes" Sep 11 15:05:26.837016 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:26.836952 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" podUID="c1c39a2a-fe09-4b71-b448-03581b6975ad" containerName="main" containerID="cri-o://487c49604709e6ecd963de04f94975313bd71d7eec3c8d38d4d7e2dd50cdec22" gracePeriod=45 Sep 11 15:05:27.000195 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:27.000162 2583 generic.go:356] "Generic (PLEG): container finished" podID="c1c39a2a-fe09-4b71-b448-03581b6975ad" containerID="487c49604709e6ecd963de04f94975313bd71d7eec3c8d38d4d7e2dd50cdec22" exitCode=0 Sep 11 15:05:27.000383 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:27.000236 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" event={"ID":"c1c39a2a-fe09-4b71-b448-03581b6975ad","Type":"ContainerDied","Data":"487c49604709e6ecd963de04f94975313bd71d7eec3c8d38d4d7e2dd50cdec22"} Sep 11 15:05:27.104020 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:27.103997 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" Sep 11 15:05:27.219894 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:27.219861 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c39a2a-fe09-4b71-b448-03581b6975ad-tls-certs\") pod \"c1c39a2a-fe09-4b71-b448-03581b6975ad\" (UID: \"c1c39a2a-fe09-4b71-b448-03581b6975ad\") " Sep 11 15:05:27.220088 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:27.219902 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5v2r\" (UniqueName: \"kubernetes.io/projected/c1c39a2a-fe09-4b71-b448-03581b6975ad-kube-api-access-l5v2r\") pod \"c1c39a2a-fe09-4b71-b448-03581b6975ad\" (UID: \"c1c39a2a-fe09-4b71-b448-03581b6975ad\") " Sep 11 15:05:27.220088 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:27.219964 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c39a2a-fe09-4b71-b448-03581b6975ad-home\") pod \"c1c39a2a-fe09-4b71-b448-03581b6975ad\" (UID: \"c1c39a2a-fe09-4b71-b448-03581b6975ad\") " Sep 11 15:05:27.220088 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:27.220058 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c39a2a-fe09-4b71-b448-03581b6975ad-tmp-dir\") pod \"c1c39a2a-fe09-4b71-b448-03581b6975ad\" (UID: \"c1c39a2a-fe09-4b71-b448-03581b6975ad\") " Sep 11 15:05:27.220275 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:27.220100 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c39a2a-fe09-4b71-b448-03581b6975ad-dshm\") pod \"c1c39a2a-fe09-4b71-b448-03581b6975ad\" (UID: \"c1c39a2a-fe09-4b71-b448-03581b6975ad\") " Sep 11 15:05:27.220275 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:27.220204 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c39a2a-fe09-4b71-b448-03581b6975ad-model-cache\") pod \"c1c39a2a-fe09-4b71-b448-03581b6975ad\" (UID: \"c1c39a2a-fe09-4b71-b448-03581b6975ad\") " Sep 11 15:05:27.220432 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:27.220280 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c39a2a-fe09-4b71-b448-03581b6975ad-home" (OuterVolumeSpecName: "home") pod "c1c39a2a-fe09-4b71-b448-03581b6975ad" (UID: "c1c39a2a-fe09-4b71-b448-03581b6975ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:05:27.220432 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:27.220408 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c39a2a-fe09-4b71-b448-03581b6975ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c1c39a2a-fe09-4b71-b448-03581b6975ad" (UID: "c1c39a2a-fe09-4b71-b448-03581b6975ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:05:27.220544 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:27.220491 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c39a2a-fe09-4b71-b448-03581b6975ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "c1c39a2a-fe09-4b71-b448-03581b6975ad" (UID: "c1c39a2a-fe09-4b71-b448-03581b6975ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:05:27.220654 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:27.220627 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c39a2a-fe09-4b71-b448-03581b6975ad-home\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:05:27.220750 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:27.220661 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c39a2a-fe09-4b71-b448-03581b6975ad-tmp-dir\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:05:27.220750 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:27.220674 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c39a2a-fe09-4b71-b448-03581b6975ad-model-cache\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:05:27.222611 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:27.222585 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1c39a2a-fe09-4b71-b448-03581b6975ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c1c39a2a-fe09-4b71-b448-03581b6975ad" (UID: "c1c39a2a-fe09-4b71-b448-03581b6975ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:05:27.222772 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:27.222752 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c39a2a-fe09-4b71-b448-03581b6975ad-dshm" (OuterVolumeSpecName: "dshm") pod "c1c39a2a-fe09-4b71-b448-03581b6975ad" (UID: "c1c39a2a-fe09-4b71-b448-03581b6975ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:05:27.222772 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:27.222759 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c39a2a-fe09-4b71-b448-03581b6975ad-kube-api-access-l5v2r" (OuterVolumeSpecName: "kube-api-access-l5v2r") pod "c1c39a2a-fe09-4b71-b448-03581b6975ad" (UID: "c1c39a2a-fe09-4b71-b448-03581b6975ad"). InnerVolumeSpecName "kube-api-access-l5v2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:05:27.321552 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:27.321514 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c39a2a-fe09-4b71-b448-03581b6975ad-tls-certs\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:05:27.321552 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:27.321549 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5v2r\" (UniqueName: \"kubernetes.io/projected/c1c39a2a-fe09-4b71-b448-03581b6975ad-kube-api-access-l5v2r\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:05:27.321552 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:27.321558 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c39a2a-fe09-4b71-b448-03581b6975ad-dshm\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:05:28.008263 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:28.008230 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" Sep 11 15:05:28.008263 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:28.008238 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7" event={"ID":"c1c39a2a-fe09-4b71-b448-03581b6975ad","Type":"ContainerDied","Data":"b364afef5d9a174786f16cd3ec731e27b247eef4893725b59488bc287f427817"} Sep 11 15:05:28.008798 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:28.008294 2583 scope.go:117] "RemoveContainer" containerID="487c49604709e6ecd963de04f94975313bd71d7eec3c8d38d4d7e2dd50cdec22" Sep 11 15:05:28.028886 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:28.028855 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7"] Sep 11 15:05:28.032437 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:28.032409 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/distinct-w0-e7ca56-kserve-54fc6f47b8-4nwj7"] Sep 11 15:05:29.331145 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:29.331111 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1c39a2a-fe09-4b71-b448-03581b6975ad" path="/var/lib/kubelet/pods/c1c39a2a-fe09-4b71-b448-03581b6975ad/volumes" Sep 11 15:05:31.409555 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:31.409519 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9"] Sep 11 15:05:31.410053 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:31.409783 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" podUID="91ab0283-f94d-45bb-9b61-ad4d927d8b68" containerName="istio-proxy" containerID="cri-o://06b129d54bd4b2ca7ca60c9f0d5c4ddaf1d4b6e90fb3368e14e6844c099934f4" gracePeriod=30 Sep 11 15:05:36.677798 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.677729 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 15:05:36.820394 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.820361 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-workload-socket\") pod \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " Sep 11 15:05:36.820571 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.820409 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istio-data\") pod \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " Sep 11 15:05:36.820571 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.820440 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istio-token\") pod \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " Sep 11 15:05:36.820571 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.820524 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istiod-ca-cert\") pod \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " Sep 11 15:05:36.820571 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.820553 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istio-envoy\") pod \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " Sep 11 15:05:36.820811 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.820578 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wc4cx\" (UniqueName: \"kubernetes.io/projected/91ab0283-f94d-45bb-9b61-ad4d927d8b68-kube-api-access-wc4cx\") pod \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " Sep 11 15:05:36.820811 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.820609 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-workload-certs\") pod \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " Sep 11 15:05:36.820811 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.820640 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istio-podinfo\") pod \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " Sep 11 15:05:36.820811 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.820671 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "91ab0283-f94d-45bb-9b61-ad4d927d8b68" (UID: "91ab0283-f94d-45bb-9b61-ad4d927d8b68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:05:36.820811 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.820690 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istio-data" (OuterVolumeSpecName: "istio-data") pod "91ab0283-f94d-45bb-9b61-ad4d927d8b68" (UID: "91ab0283-f94d-45bb-9b61-ad4d927d8b68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:05:36.820811 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.820683 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-credential-socket\") pod \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\" (UID: \"91ab0283-f94d-45bb-9b61-ad4d927d8b68\") " Sep 11 15:05:36.821115 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.820953 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "91ab0283-f94d-45bb-9b61-ad4d927d8b68" (UID: "91ab0283-f94d-45bb-9b61-ad4d927d8b68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:05:36.821115 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.821081 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "91ab0283-f94d-45bb-9b61-ad4d927d8b68" (UID: "91ab0283-f94d-45bb-9b61-ad4d927d8b68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:05:36.821275 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.821251 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-credential-socket\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:05:36.821406 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.821279 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-workload-socket\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:05:36.821406 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.821294 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istio-data\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:05:36.821406 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.821307 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-workload-certs\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:05:36.821406 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.821376 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "91ab0283-f94d-45bb-9b61-ad4d927d8b68" (UID: "91ab0283-f94d-45bb-9b61-ad4d927d8b68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:05:36.822976 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.822944 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "91ab0283-f94d-45bb-9b61-ad4d927d8b68" (UID: "91ab0283-f94d-45bb-9b61-ad4d927d8b68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:05:36.823099 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.822996 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "91ab0283-f94d-45bb-9b61-ad4d927d8b68" (UID: "91ab0283-f94d-45bb-9b61-ad4d927d8b68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 15:05:36.823186 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.823166 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istio-token" (OuterVolumeSpecName: "istio-token") pod "91ab0283-f94d-45bb-9b61-ad4d927d8b68" (UID: "91ab0283-f94d-45bb-9b61-ad4d927d8b68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:05:36.823248 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.823193 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ab0283-f94d-45bb-9b61-ad4d927d8b68-kube-api-access-wc4cx" (OuterVolumeSpecName: "kube-api-access-wc4cx") pod "91ab0283-f94d-45bb-9b61-ad4d927d8b68" (UID: "91ab0283-f94d-45bb-9b61-ad4d927d8b68"). InnerVolumeSpecName "kube-api-access-wc4cx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:05:36.921867 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.921833 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istio-token\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:05:36.921867 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.921862 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istiod-ca-cert\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:05:36.921867 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.921871 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istio-envoy\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:05:36.922116 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.921881 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc4cx\" (UniqueName: \"kubernetes.io/projected/91ab0283-f94d-45bb-9b61-ad4d927d8b68-kube-api-access-wc4cx\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:05:36.922116 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:36.921892 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ab0283-f94d-45bb-9b61-ad4d927d8b68-istio-podinfo\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:05:37.067472 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:37.067378 2583 generic.go:356] "Generic (PLEG): container finished" podID="91ab0283-f94d-45bb-9b61-ad4d927d8b68" containerID="06b129d54bd4b2ca7ca60c9f0d5c4ddaf1d4b6e90fb3368e14e6844c099934f4" exitCode=0 Sep 11 15:05:37.067472 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:37.067437 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" Sep 11 15:05:37.067472 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:37.067461 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" event={"ID":"91ab0283-f94d-45bb-9b61-ad4d927d8b68","Type":"ContainerDied","Data":"06b129d54bd4b2ca7ca60c9f0d5c4ddaf1d4b6e90fb3368e14e6844c099934f4"} Sep 11 15:05:37.067792 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:37.067542 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9" event={"ID":"91ab0283-f94d-45bb-9b61-ad4d927d8b68","Type":"ContainerDied","Data":"4e75b77f3dabcf5ff279d85ddab069aa8a1b4657c8fba1afe0668789df19244a"} Sep 11 15:05:37.067792 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:37.067565 2583 scope.go:117] "RemoveContainer" containerID="06b129d54bd4b2ca7ca60c9f0d5c4ddaf1d4b6e90fb3368e14e6844c099934f4" Sep 11 15:05:37.078524 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:37.078507 2583 scope.go:117] "RemoveContainer" containerID="315a9c33c41e8e74ae348e51248b8be0e053a6fd86fc945fd0f45cd81ebe646f" Sep 11 15:05:37.088139 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:37.088117 2583 scope.go:117] "RemoveContainer" containerID="06b129d54bd4b2ca7ca60c9f0d5c4ddaf1d4b6e90fb3368e14e6844c099934f4" Sep 11 15:05:37.088413 ip-10-0-135-39 kubenswrapper[2583]: E0911 15:05:37.088394 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06b129d54bd4b2ca7ca60c9f0d5c4ddaf1d4b6e90fb3368e14e6844c099934f4\": container with ID starting with 06b129d54bd4b2ca7ca60c9f0d5c4ddaf1d4b6e90fb3368e14e6844c099934f4 not found: ID does not exist" containerID="06b129d54bd4b2ca7ca60c9f0d5c4ddaf1d4b6e90fb3368e14e6844c099934f4" Sep 11 15:05:37.088467 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:37.088421 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b129d54bd4b2ca7ca60c9f0d5c4ddaf1d4b6e90fb3368e14e6844c099934f4"} err="failed to get container status \"06b129d54bd4b2ca7ca60c9f0d5c4ddaf1d4b6e90fb3368e14e6844c099934f4\": rpc error: code = NotFound desc = could not find container \"06b129d54bd4b2ca7ca60c9f0d5c4ddaf1d4b6e90fb3368e14e6844c099934f4\": container with ID starting with 06b129d54bd4b2ca7ca60c9f0d5c4ddaf1d4b6e90fb3368e14e6844c099934f4 not found: ID does not exist" Sep 11 15:05:37.088467 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:37.088440 2583 scope.go:117] "RemoveContainer" containerID="315a9c33c41e8e74ae348e51248b8be0e053a6fd86fc945fd0f45cd81ebe646f" Sep 11 15:05:37.088733 ip-10-0-135-39 kubenswrapper[2583]: E0911 15:05:37.088690 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"315a9c33c41e8e74ae348e51248b8be0e053a6fd86fc945fd0f45cd81ebe646f\": container with ID starting with 315a9c33c41e8e74ae348e51248b8be0e053a6fd86fc945fd0f45cd81ebe646f not found: ID does not exist" containerID="315a9c33c41e8e74ae348e51248b8be0e053a6fd86fc945fd0f45cd81ebe646f" Sep 11 15:05:37.088805 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:37.088741 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"315a9c33c41e8e74ae348e51248b8be0e053a6fd86fc945fd0f45cd81ebe646f"} err="failed to get container status \"315a9c33c41e8e74ae348e51248b8be0e053a6fd86fc945fd0f45cd81ebe646f\": rpc error: code = NotFound desc = could not find container \"315a9c33c41e8e74ae348e51248b8be0e053a6fd86fc945fd0f45cd81ebe646f\": container with ID starting with 315a9c33c41e8e74ae348e51248b8be0e053a6fd86fc945fd0f45cd81ebe646f not found: ID does not exist" Sep 11 15:05:37.090409 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:37.090388 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9"] Sep 11 15:05:37.094060 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:37.094040 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e7ca56-openshift-default-78999d97db-5n8h9"] Sep 11 15:05:37.331886 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:05:37.331804 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91ab0283-f94d-45bb-9b61-ad4d927d8b68" path="/var/lib/kubelet/pods/91ab0283-f94d-45bb-9b61-ad4d927d8b68/volumes" Sep 11 15:06:06.959192 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:06.959148 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54b7f9-k2wkx"] Sep 11 15:06:06.959823 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:06.959795 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31b708e9-eb7c-47ee-b177-78c6bc3b97b9" containerName="revoke-keys" Sep 11 15:06:06.959823 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:06.959816 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="31b708e9-eb7c-47ee-b177-78c6bc3b97b9" containerName="revoke-keys" Sep 11 15:06:06.959962 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:06.959833 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1c39a2a-fe09-4b71-b448-03581b6975ad" containerName="main" Sep 11 15:06:06.959962 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:06.959841 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1c39a2a-fe09-4b71-b448-03581b6975ad" containerName="main" Sep 11 15:06:06.959962 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:06.959860 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ab0283-f94d-45bb-9b61-ad4d927d8b68" containerName="istio-proxy" Sep 11 15:06:06.959962 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:06.959870 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ab0283-f94d-45bb-9b61-ad4d927d8b68" containerName="istio-proxy" Sep 11 15:06:06.959962 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:06.959887 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="743de4e7-8ce3-4871-9d50-6f05b34bb27b" containerName="payload-pre-processing" Sep 11 15:06:06.959962 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:06.959895 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="743de4e7-8ce3-4871-9d50-6f05b34bb27b" containerName="payload-pre-processing" Sep 11 15:06:06.959962 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:06.959910 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ab0283-f94d-45bb-9b61-ad4d927d8b68" containerName="istio-proxy" Sep 11 15:06:06.959962 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:06.959918 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ab0283-f94d-45bb-9b61-ad4d927d8b68" containerName="istio-proxy" Sep 11 15:06:06.960340 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:06.960028 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="31b708e9-eb7c-47ee-b177-78c6bc3b97b9" containerName="revoke-keys" Sep 11 15:06:06.960340 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:06.960043 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="91ab0283-f94d-45bb-9b61-ad4d927d8b68" containerName="istio-proxy" Sep 11 15:06:06.960340 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:06.960053 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="743de4e7-8ce3-4871-9d50-6f05b34bb27b" containerName="payload-pre-processing" Sep 11 15:06:06.960340 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:06.960065 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1c39a2a-fe09-4b71-b448-03581b6975ad" containerName="main" Sep 11 15:06:06.963518 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:06.963496 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54b7f9-k2wkx" Sep 11 15:06:06.965992 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:06.965918 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:06:06.966116 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:06.966002 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nks8w\"" Sep 11 15:06:06.974901 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:06.974878 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54b7f9-k2wkx"] Sep 11 15:06:07.124186 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:07.124149 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e81d2ab-8267-46a6-a889-2d799abd4d50-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-54b7f9-k2wkx\" (UID: \"0e81d2ab-8267-46a6-a889-2d799abd4d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54b7f9-k2wkx" Sep 11 15:06:07.225533 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:07.225444 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e81d2ab-8267-46a6-a889-2d799abd4d50-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-54b7f9-k2wkx\" (UID: \"0e81d2ab-8267-46a6-a889-2d799abd4d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54b7f9-k2wkx" Sep 11 15:06:07.226085 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:07.226064 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e81d2ab-8267-46a6-a889-2d799abd4d50-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-54b7f9-k2wkx\" (UID: \"0e81d2ab-8267-46a6-a889-2d799abd4d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54b7f9-k2wkx" Sep 11 15:06:07.274932 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:07.274897 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54b7f9-k2wkx" Sep 11 15:06:07.548808 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:07.548780 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54b7f9-k2wkx"] Sep 11 15:06:07.550168 ip-10-0-135-39 kubenswrapper[2583]: W0911 15:06:07.550143 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e81d2ab_8267_46a6_a889_2d799abd4d50.slice/crio-ce9803f41728eec4a7a0bbdf20ca53f4540a10b459951c17d472c75073c74f8e WatchSource:0}: Error finding container ce9803f41728eec4a7a0bbdf20ca53f4540a10b459951c17d472c75073c74f8e: Status 404 returned error can't find the container with id ce9803f41728eec4a7a0bbdf20ca53f4540a10b459951c17d472c75073c74f8e Sep 11 15:06:08.264425 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:08.264390 2583 generic.go:356] "Generic (PLEG): container finished" podID="0e81d2ab-8267-46a6-a889-2d799abd4d50" containerID="5fe9812e427d6b2eacc9c7dd0272497eebdd3649b807060ef75e2889d74167ba" exitCode=0 Sep 11 15:06:08.264826 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:08.264470 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54b7f9-k2wkx" event={"ID":"0e81d2ab-8267-46a6-a889-2d799abd4d50","Type":"ContainerDied","Data":"5fe9812e427d6b2eacc9c7dd0272497eebdd3649b807060ef75e2889d74167ba"} Sep 11 15:06:08.264826 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:08.264528 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54b7f9-k2wkx" event={"ID":"0e81d2ab-8267-46a6-a889-2d799abd4d50","Type":"ContainerStarted","Data":"ce9803f41728eec4a7a0bbdf20ca53f4540a10b459951c17d472c75073c74f8e"} Sep 11 15:06:09.429194 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:09.429164 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54b7f9-k2wkx" Sep 11 15:06:09.540700 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:09.540614 2583 scope.go:117] "RemoveContainer" containerID="f9d2f99d0e7fd2731048cea2da615f5d5457840d92229991f31b40a8d1294de1" Sep 11 15:06:09.551989 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:09.551964 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e81d2ab-8267-46a6-a889-2d799abd4d50-maas-api-service-ca\") pod \"0e81d2ab-8267-46a6-a889-2d799abd4d50\" (UID: \"0e81d2ab-8267-46a6-a889-2d799abd4d50\") " Sep 11 15:06:09.552298 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:09.552276 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e81d2ab-8267-46a6-a889-2d799abd4d50-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0e81d2ab-8267-46a6-a889-2d799abd4d50" (UID: "0e81d2ab-8267-46a6-a889-2d799abd4d50"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:06:09.552760 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:09.552744 2583 scope.go:117] "RemoveContainer" containerID="1ed88ff43821c65b6ff93b59572385e601b160811f6972268cdc626fa7b46c09" Sep 11 15:06:09.561181 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:09.561159 2583 scope.go:117] "RemoveContainer" containerID="967a4ea78e000f6ca3b3c21fa6994bcc85cd04b965d8677cf09e83b71a22ebcb" Sep 11 15:06:09.653403 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:09.653370 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e81d2ab-8267-46a6-a889-2d799abd4d50-maas-api-service-ca\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:06:10.280068 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:10.280040 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54b7f9-k2wkx" Sep 11 15:06:10.280269 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:10.280044 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54b7f9-k2wkx" event={"ID":"0e81d2ab-8267-46a6-a889-2d799abd4d50","Type":"ContainerDied","Data":"ce9803f41728eec4a7a0bbdf20ca53f4540a10b459951c17d472c75073c74f8e"} Sep 11 15:06:10.280269 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:10.280176 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce9803f41728eec4a7a0bbdf20ca53f4540a10b459951c17d472c75073c74f8e" Sep 11 15:06:15.964628 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:15.964588 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-837632-ncxrj"] Sep 11 15:06:15.965070 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:15.965055 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e81d2ab-8267-46a6-a889-2d799abd4d50" containerName="revoke-keys" Sep 11 15:06:15.965070 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:15.965071 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e81d2ab-8267-46a6-a889-2d799abd4d50" containerName="revoke-keys" Sep 11 15:06:15.965161 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:15.965151 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="91ab0283-f94d-45bb-9b61-ad4d927d8b68" containerName="istio-proxy" Sep 11 15:06:15.965205 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:15.965164 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e81d2ab-8267-46a6-a889-2d799abd4d50" containerName="revoke-keys" Sep 11 15:06:15.968245 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:15.968226 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-837632-ncxrj" Sep 11 15:06:15.976826 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:15.976796 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:06:15.977070 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:15.977030 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nks8w\"" Sep 11 15:06:15.981359 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:15.981324 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-837632-ncxrj"] Sep 11 15:06:16.011285 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:16.011250 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb060917-8fd7-4b9b-8d1c-bcc7ffa0b274-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-837632-ncxrj\" (UID: \"eb060917-8fd7-4b9b-8d1c-bcc7ffa0b274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-837632-ncxrj" Sep 11 15:06:16.111789 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:16.111736 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb060917-8fd7-4b9b-8d1c-bcc7ffa0b274-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-837632-ncxrj\" (UID: \"eb060917-8fd7-4b9b-8d1c-bcc7ffa0b274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-837632-ncxrj" Sep 11 15:06:16.112322 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:16.112302 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb060917-8fd7-4b9b-8d1c-bcc7ffa0b274-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-837632-ncxrj\" (UID: \"eb060917-8fd7-4b9b-8d1c-bcc7ffa0b274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-837632-ncxrj" Sep 11 15:06:16.285882 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:16.285789 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-837632-ncxrj" Sep 11 15:06:16.569674 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:16.569642 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-837632-ncxrj"] Sep 11 15:06:16.570911 ip-10-0-135-39 kubenswrapper[2583]: W0911 15:06:16.570880 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb060917_8fd7_4b9b_8d1c_bcc7ffa0b274.slice/crio-ec742cdf88b13627d4d9d4050861fc26b4aec2803b1bed49cf8ba32e8201ba6c WatchSource:0}: Error finding container ec742cdf88b13627d4d9d4050861fc26b4aec2803b1bed49cf8ba32e8201ba6c: Status 404 returned error can't find the container with id ec742cdf88b13627d4d9d4050861fc26b4aec2803b1bed49cf8ba32e8201ba6c Sep 11 15:06:17.325589 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:17.325555 2583 generic.go:356] "Generic (PLEG): container finished" podID="eb060917-8fd7-4b9b-8d1c-bcc7ffa0b274" containerID="b993d9b7a64de20f8edce937df1b8e65ae5984b8f639a3a40cf3685fbaf40f82" exitCode=0 Sep 11 15:06:17.330678 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:17.330643 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-837632-ncxrj" event={"ID":"eb060917-8fd7-4b9b-8d1c-bcc7ffa0b274","Type":"ContainerDied","Data":"b993d9b7a64de20f8edce937df1b8e65ae5984b8f639a3a40cf3685fbaf40f82"} Sep 11 15:06:17.330828 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:17.330697 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-837632-ncxrj" event={"ID":"eb060917-8fd7-4b9b-8d1c-bcc7ffa0b274","Type":"ContainerStarted","Data":"ec742cdf88b13627d4d9d4050861fc26b4aec2803b1bed49cf8ba32e8201ba6c"} Sep 11 15:06:18.471507 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:18.471476 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-837632-ncxrj" Sep 11 15:06:18.533447 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:18.533414 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb060917-8fd7-4b9b-8d1c-bcc7ffa0b274-maas-api-service-ca\") pod \"eb060917-8fd7-4b9b-8d1c-bcc7ffa0b274\" (UID: \"eb060917-8fd7-4b9b-8d1c-bcc7ffa0b274\") " Sep 11 15:06:18.533766 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:18.533744 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb060917-8fd7-4b9b-8d1c-bcc7ffa0b274-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eb060917-8fd7-4b9b-8d1c-bcc7ffa0b274" (UID: "eb060917-8fd7-4b9b-8d1c-bcc7ffa0b274"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:06:18.634136 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:18.634106 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb060917-8fd7-4b9b-8d1c-bcc7ffa0b274-maas-api-service-ca\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:06:19.339272 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:19.339241 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-837632-ncxrj" Sep 11 15:06:19.339453 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:19.339268 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-837632-ncxrj" event={"ID":"eb060917-8fd7-4b9b-8d1c-bcc7ffa0b274","Type":"ContainerDied","Data":"ec742cdf88b13627d4d9d4050861fc26b4aec2803b1bed49cf8ba32e8201ba6c"} Sep 11 15:06:19.339453 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:19.339316 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec742cdf88b13627d4d9d4050861fc26b4aec2803b1bed49cf8ba32e8201ba6c" Sep 11 15:06:20.822989 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:20.822949 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7"] Sep 11 15:06:20.832800 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:20.832764 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk"] Sep 11 15:06:22.798333 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:22.798300 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh"] Sep 11 15:06:22.798786 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:22.798525 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh" podUID="b0b07e34-b488-460f-abf9-45dd0c2e0387" containerName="payload-processing" containerID="cri-o://4defc6f7f82c686e7d37b5cb9a84ac3d2287fb0aa864e725d6d2c0972a670d80" gracePeriod=30 Sep 11 15:06:22.988666 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:22.988633 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54b7f9-k2wkx"] Sep 11 15:06:22.991385 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:22.991356 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-54b7f9-k2wkx"] Sep 11 15:06:23.058249 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:23.058181 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh" Sep 11 15:06:23.072028 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:23.071991 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7b85h\" (UniqueName: \"kubernetes.io/projected/b0b07e34-b488-460f-abf9-45dd0c2e0387-kube-api-access-7b85h\") pod \"b0b07e34-b488-460f-abf9-45dd0c2e0387\" (UID: \"b0b07e34-b488-460f-abf9-45dd0c2e0387\") " Sep 11 15:06:23.072187 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:23.072040 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0b07e34-b488-460f-abf9-45dd0c2e0387-plugins-config-volume\") pod \"b0b07e34-b488-460f-abf9-45dd0c2e0387\" (UID: \"b0b07e34-b488-460f-abf9-45dd0c2e0387\") " Sep 11 15:06:23.072459 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:23.072437 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0b07e34-b488-460f-abf9-45dd0c2e0387-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b0b07e34-b488-460f-abf9-45dd0c2e0387" (UID: "b0b07e34-b488-460f-abf9-45dd0c2e0387"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:06:23.074081 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:23.074056 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b07e34-b488-460f-abf9-45dd0c2e0387-kube-api-access-7b85h" (OuterVolumeSpecName: "kube-api-access-7b85h") pod "b0b07e34-b488-460f-abf9-45dd0c2e0387" (UID: "b0b07e34-b488-460f-abf9-45dd0c2e0387"). InnerVolumeSpecName "kube-api-access-7b85h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:06:23.173202 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:23.173172 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b85h\" (UniqueName: \"kubernetes.io/projected/b0b07e34-b488-460f-abf9-45dd0c2e0387-kube-api-access-7b85h\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:06:23.173202 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:23.173199 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0b07e34-b488-460f-abf9-45dd0c2e0387-plugins-config-volume\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:06:23.330647 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:23.330562 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e81d2ab-8267-46a6-a889-2d799abd4d50" path="/var/lib/kubelet/pods/0e81d2ab-8267-46a6-a889-2d799abd4d50/volumes" Sep 11 15:06:23.364633 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:23.364604 2583 generic.go:356] "Generic (PLEG): container finished" podID="b0b07e34-b488-460f-abf9-45dd0c2e0387" containerID="4defc6f7f82c686e7d37b5cb9a84ac3d2287fb0aa864e725d6d2c0972a670d80" exitCode=0 Sep 11 15:06:23.364776 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:23.364650 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh" Sep 11 15:06:23.364776 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:23.364695 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh" event={"ID":"b0b07e34-b488-460f-abf9-45dd0c2e0387","Type":"ContainerDied","Data":"4defc6f7f82c686e7d37b5cb9a84ac3d2287fb0aa864e725d6d2c0972a670d80"} Sep 11 15:06:23.364776 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:23.364773 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh" event={"ID":"b0b07e34-b488-460f-abf9-45dd0c2e0387","Type":"ContainerDied","Data":"96d52e65b6883845297ff7a94db200ff7c84952bfccd08dec0b965aa56d299ed"} Sep 11 15:06:23.364965 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:23.364805 2583 scope.go:117] "RemoveContainer" containerID="4defc6f7f82c686e7d37b5cb9a84ac3d2287fb0aa864e725d6d2c0972a670d80" Sep 11 15:06:23.374528 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:23.374502 2583 scope.go:117] "RemoveContainer" containerID="4defc6f7f82c686e7d37b5cb9a84ac3d2287fb0aa864e725d6d2c0972a670d80" Sep 11 15:06:23.374833 ip-10-0-135-39 kubenswrapper[2583]: E0911 15:06:23.374806 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4defc6f7f82c686e7d37b5cb9a84ac3d2287fb0aa864e725d6d2c0972a670d80\": container with ID starting with 4defc6f7f82c686e7d37b5cb9a84ac3d2287fb0aa864e725d6d2c0972a670d80 not found: ID does not exist" containerID="4defc6f7f82c686e7d37b5cb9a84ac3d2287fb0aa864e725d6d2c0972a670d80" Sep 11 15:06:23.374920 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:23.374845 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4defc6f7f82c686e7d37b5cb9a84ac3d2287fb0aa864e725d6d2c0972a670d80"} err="failed to get container status \"4defc6f7f82c686e7d37b5cb9a84ac3d2287fb0aa864e725d6d2c0972a670d80\": rpc error: code = NotFound desc = could not find container \"4defc6f7f82c686e7d37b5cb9a84ac3d2287fb0aa864e725d6d2c0972a670d80\": container with ID starting with 4defc6f7f82c686e7d37b5cb9a84ac3d2287fb0aa864e725d6d2c0972a670d80 not found: ID does not exist" Sep 11 15:06:23.390597 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:23.390573 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh"] Sep 11 15:06:23.396402 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:23.396380 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-54b7f9-74df86bfc-4cphh"] Sep 11 15:06:25.331198 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:25.331167 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0b07e34-b488-460f-abf9-45dd0c2e0387" path="/var/lib/kubelet/pods/b0b07e34-b488-460f-abf9-45dd0c2e0387/volumes" Sep 11 15:06:30.560860 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:30.560822 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14a808-jw94p"] Sep 11 15:06:30.561231 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:30.561222 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0b07e34-b488-460f-abf9-45dd0c2e0387" containerName="payload-processing" Sep 11 15:06:30.561286 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:30.561232 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0b07e34-b488-460f-abf9-45dd0c2e0387" containerName="payload-processing" Sep 11 15:06:30.561286 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:30.561242 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb060917-8fd7-4b9b-8d1c-bcc7ffa0b274" containerName="revoke-keys" Sep 11 15:06:30.561286 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:30.561247 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb060917-8fd7-4b9b-8d1c-bcc7ffa0b274" containerName="revoke-keys" Sep 11 15:06:30.561386 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:30.561322 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0b07e34-b488-460f-abf9-45dd0c2e0387" containerName="payload-processing" Sep 11 15:06:30.561386 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:30.561334 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb060917-8fd7-4b9b-8d1c-bcc7ffa0b274" containerName="revoke-keys" Sep 11 15:06:30.566251 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:30.566232 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14a808-jw94p" Sep 11 15:06:30.568816 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:30.568788 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:06:30.569530 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:30.569508 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nks8w\"" Sep 11 15:06:30.573000 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:30.572980 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14a808-jw94p"] Sep 11 15:06:30.640045 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:30.640008 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/498da86c-163a-44ec-a573-0fe185893871-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-14a808-jw94p\" (UID: \"498da86c-163a-44ec-a573-0fe185893871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14a808-jw94p" Sep 11 15:06:30.740668 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:30.740628 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/498da86c-163a-44ec-a573-0fe185893871-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-14a808-jw94p\" (UID: \"498da86c-163a-44ec-a573-0fe185893871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14a808-jw94p" Sep 11 15:06:30.741285 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:30.741265 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/498da86c-163a-44ec-a573-0fe185893871-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-14a808-jw94p\" (UID: \"498da86c-163a-44ec-a573-0fe185893871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14a808-jw94p" Sep 11 15:06:30.877582 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:30.877552 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14a808-jw94p" Sep 11 15:06:31.168098 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:31.168070 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14a808-jw94p"] Sep 11 15:06:31.169873 ip-10-0-135-39 kubenswrapper[2583]: W0911 15:06:31.169842 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod498da86c_163a_44ec_a573_0fe185893871.slice/crio-2b6f68fe06ff41ed647a1206725443a45c614ae57e00c6673124e6d6679fedb2 WatchSource:0}: Error finding container 2b6f68fe06ff41ed647a1206725443a45c614ae57e00c6673124e6d6679fedb2: Status 404 returned error can't find the container with id 2b6f68fe06ff41ed647a1206725443a45c614ae57e00c6673124e6d6679fedb2 Sep 11 15:06:31.419993 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:31.419898 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14a808-jw94p" event={"ID":"498da86c-163a-44ec-a573-0fe185893871","Type":"ContainerStarted","Data":"2b6f68fe06ff41ed647a1206725443a45c614ae57e00c6673124e6d6679fedb2"} Sep 11 15:06:32.428129 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:32.428088 2583 generic.go:356] "Generic (PLEG): container finished" podID="498da86c-163a-44ec-a573-0fe185893871" containerID="327825ae8e320cd5fb6beb3a50c9afdc703b3530361968893e40018296b3f4e8" exitCode=0 Sep 11 15:06:32.428524 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:32.428147 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14a808-jw94p" event={"ID":"498da86c-163a-44ec-a573-0fe185893871","Type":"ContainerDied","Data":"327825ae8e320cd5fb6beb3a50c9afdc703b3530361968893e40018296b3f4e8"} Sep 11 15:06:33.073113 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:33.073077 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-837632-ncxrj"] Sep 11 15:06:33.076121 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:33.076095 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-837632-ncxrj"] Sep 11 15:06:33.330970 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:33.330880 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb060917-8fd7-4b9b-8d1c-bcc7ffa0b274" path="/var/lib/kubelet/pods/eb060917-8fd7-4b9b-8d1c-bcc7ffa0b274/volumes" Sep 11 15:06:33.578798 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:33.578771 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14a808-jw94p" Sep 11 15:06:33.671604 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:33.671574 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/498da86c-163a-44ec-a573-0fe185893871-maas-api-service-ca\") pod \"498da86c-163a-44ec-a573-0fe185893871\" (UID: \"498da86c-163a-44ec-a573-0fe185893871\") " Sep 11 15:06:33.671957 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:33.671934 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/498da86c-163a-44ec-a573-0fe185893871-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "498da86c-163a-44ec-a573-0fe185893871" (UID: "498da86c-163a-44ec-a573-0fe185893871"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:06:33.773208 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:33.773172 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/498da86c-163a-44ec-a573-0fe185893871-maas-api-service-ca\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:06:34.440859 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:34.440825 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14a808-jw94p" event={"ID":"498da86c-163a-44ec-a573-0fe185893871","Type":"ContainerDied","Data":"2b6f68fe06ff41ed647a1206725443a45c614ae57e00c6673124e6d6679fedb2"} Sep 11 15:06:34.440859 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:34.440845 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14a808-jw94p" Sep 11 15:06:34.441077 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:34.440872 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b6f68fe06ff41ed647a1206725443a45c614ae57e00c6673124e6d6679fedb2" Sep 11 15:06:35.842424 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:35.842376 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" podUID="dbd638a4-11fa-4a78-ac08-351a0e415f27" containerName="main" containerID="cri-o://47f96d0a9f5790586bf4a2d82cd9d67821b33aa3cb29948af726121954bc5837" gracePeriod=45 Sep 11 15:06:35.852734 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:35.852683 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" podUID="b3f7c8b2-e74b-4902-a002-104b20fac586" containerName="main" containerID="cri-o://2529b55b23417c5818b4d99ba52e5e80586767bbbe67a1fe024480725de6f9d2" gracePeriod=45 Sep 11 15:06:36.137672 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.137642 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" Sep 11 15:06:36.149232 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.149204 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" Sep 11 15:06:36.196120 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.196081 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3f7c8b2-e74b-4902-a002-104b20fac586-home\") pod \"b3f7c8b2-e74b-4902-a002-104b20fac586\" (UID: \"b3f7c8b2-e74b-4902-a002-104b20fac586\") " Sep 11 15:06:36.196313 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.196138 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wm4l\" (UniqueName: \"kubernetes.io/projected/b3f7c8b2-e74b-4902-a002-104b20fac586-kube-api-access-2wm4l\") pod \"b3f7c8b2-e74b-4902-a002-104b20fac586\" (UID: \"b3f7c8b2-e74b-4902-a002-104b20fac586\") " Sep 11 15:06:36.196313 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.196179 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbd638a4-11fa-4a78-ac08-351a0e415f27-home\") pod \"dbd638a4-11fa-4a78-ac08-351a0e415f27\" (UID: \"dbd638a4-11fa-4a78-ac08-351a0e415f27\") " Sep 11 15:06:36.196313 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.196210 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbd638a4-11fa-4a78-ac08-351a0e415f27-tls-certs\") pod \"dbd638a4-11fa-4a78-ac08-351a0e415f27\" (UID: \"dbd638a4-11fa-4a78-ac08-351a0e415f27\") " Sep 11 15:06:36.196485 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.196407 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3f7c8b2-e74b-4902-a002-104b20fac586-home" (OuterVolumeSpecName: "home") pod "b3f7c8b2-e74b-4902-a002-104b20fac586" (UID: "b3f7c8b2-e74b-4902-a002-104b20fac586"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:06:36.196485 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.196428 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd638a4-11fa-4a78-ac08-351a0e415f27-home" (OuterVolumeSpecName: "home") pod "dbd638a4-11fa-4a78-ac08-351a0e415f27" (UID: "dbd638a4-11fa-4a78-ac08-351a0e415f27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:06:36.196594 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.196502 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbd638a4-11fa-4a78-ac08-351a0e415f27-model-cache\") pod \"dbd638a4-11fa-4a78-ac08-351a0e415f27\" (UID: \"dbd638a4-11fa-4a78-ac08-351a0e415f27\") " Sep 11 15:06:36.196594 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.196542 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbd638a4-11fa-4a78-ac08-351a0e415f27-dshm\") pod \"dbd638a4-11fa-4a78-ac08-351a0e415f27\" (UID: \"dbd638a4-11fa-4a78-ac08-351a0e415f27\") " Sep 11 15:06:36.196594 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.196584 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbd638a4-11fa-4a78-ac08-351a0e415f27-tmp-dir\") pod \"dbd638a4-11fa-4a78-ac08-351a0e415f27\" (UID: \"dbd638a4-11fa-4a78-ac08-351a0e415f27\") " Sep 11 15:06:36.196807 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.196648 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7v7r\" (UniqueName: \"kubernetes.io/projected/dbd638a4-11fa-4a78-ac08-351a0e415f27-kube-api-access-z7v7r\") pod \"dbd638a4-11fa-4a78-ac08-351a0e415f27\" (UID: \"dbd638a4-11fa-4a78-ac08-351a0e415f27\") " Sep 11 15:06:36.196807 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.196698 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd638a4-11fa-4a78-ac08-351a0e415f27-model-cache" (OuterVolumeSpecName: "model-cache") pod "dbd638a4-11fa-4a78-ac08-351a0e415f27" (UID: "dbd638a4-11fa-4a78-ac08-351a0e415f27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:06:36.196807 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.196793 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3f7c8b2-e74b-4902-a002-104b20fac586-tmp-dir\") pod \"b3f7c8b2-e74b-4902-a002-104b20fac586\" (UID: \"b3f7c8b2-e74b-4902-a002-104b20fac586\") " Sep 11 15:06:36.196973 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.196837 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3f7c8b2-e74b-4902-a002-104b20fac586-tls-certs\") pod \"b3f7c8b2-e74b-4902-a002-104b20fac586\" (UID: \"b3f7c8b2-e74b-4902-a002-104b20fac586\") " Sep 11 15:06:36.196973 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.196865 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3f7c8b2-e74b-4902-a002-104b20fac586-dshm\") pod \"b3f7c8b2-e74b-4902-a002-104b20fac586\" (UID: \"b3f7c8b2-e74b-4902-a002-104b20fac586\") " Sep 11 15:06:36.196973 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.196897 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3f7c8b2-e74b-4902-a002-104b20fac586-model-cache\") pod \"b3f7c8b2-e74b-4902-a002-104b20fac586\" (UID: \"b3f7c8b2-e74b-4902-a002-104b20fac586\") " Sep 11 15:06:36.197234 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.197217 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3f7c8b2-e74b-4902-a002-104b20fac586-home\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:06:36.197310 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.197230 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3f7c8b2-e74b-4902-a002-104b20fac586-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b3f7c8b2-e74b-4902-a002-104b20fac586" (UID: "b3f7c8b2-e74b-4902-a002-104b20fac586"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:06:36.197310 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.197238 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbd638a4-11fa-4a78-ac08-351a0e415f27-home\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:06:36.197310 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.197276 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbd638a4-11fa-4a78-ac08-351a0e415f27-model-cache\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:06:36.197482 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.197423 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3f7c8b2-e74b-4902-a002-104b20fac586-model-cache" (OuterVolumeSpecName: "model-cache") pod "b3f7c8b2-e74b-4902-a002-104b20fac586" (UID: "b3f7c8b2-e74b-4902-a002-104b20fac586"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:06:36.197538 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.197483 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd638a4-11fa-4a78-ac08-351a0e415f27-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dbd638a4-11fa-4a78-ac08-351a0e415f27" (UID: "dbd638a4-11fa-4a78-ac08-351a0e415f27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:06:36.198839 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.198801 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbd638a4-11fa-4a78-ac08-351a0e415f27-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dbd638a4-11fa-4a78-ac08-351a0e415f27" (UID: "dbd638a4-11fa-4a78-ac08-351a0e415f27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:06:36.198990 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.198941 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3f7c8b2-e74b-4902-a002-104b20fac586-kube-api-access-2wm4l" (OuterVolumeSpecName: "kube-api-access-2wm4l") pod "b3f7c8b2-e74b-4902-a002-104b20fac586" (UID: "b3f7c8b2-e74b-4902-a002-104b20fac586"). InnerVolumeSpecName "kube-api-access-2wm4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:06:36.199250 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.199231 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd638a4-11fa-4a78-ac08-351a0e415f27-dshm" (OuterVolumeSpecName: "dshm") pod "dbd638a4-11fa-4a78-ac08-351a0e415f27" (UID: "dbd638a4-11fa-4a78-ac08-351a0e415f27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:06:36.199883 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.199857 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3f7c8b2-e74b-4902-a002-104b20fac586-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b3f7c8b2-e74b-4902-a002-104b20fac586" (UID: "b3f7c8b2-e74b-4902-a002-104b20fac586"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:06:36.200002 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.199988 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3f7c8b2-e74b-4902-a002-104b20fac586-dshm" (OuterVolumeSpecName: "dshm") pod "b3f7c8b2-e74b-4902-a002-104b20fac586" (UID: "b3f7c8b2-e74b-4902-a002-104b20fac586"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:06:36.200300 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.200276 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd638a4-11fa-4a78-ac08-351a0e415f27-kube-api-access-z7v7r" (OuterVolumeSpecName: "kube-api-access-z7v7r") pod "dbd638a4-11fa-4a78-ac08-351a0e415f27" (UID: "dbd638a4-11fa-4a78-ac08-351a0e415f27"). InnerVolumeSpecName "kube-api-access-z7v7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:06:36.298186 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.298155 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3f7c8b2-e74b-4902-a002-104b20fac586-tmp-dir\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:06:36.298186 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.298183 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3f7c8b2-e74b-4902-a002-104b20fac586-tls-certs\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:06:36.298186 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.298192 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3f7c8b2-e74b-4902-a002-104b20fac586-dshm\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:06:36.298186 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.298202 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3f7c8b2-e74b-4902-a002-104b20fac586-model-cache\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:06:36.298458 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.298215 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wm4l\" (UniqueName: \"kubernetes.io/projected/b3f7c8b2-e74b-4902-a002-104b20fac586-kube-api-access-2wm4l\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:06:36.298458 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.298224 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbd638a4-11fa-4a78-ac08-351a0e415f27-tls-certs\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:06:36.298458 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.298234 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbd638a4-11fa-4a78-ac08-351a0e415f27-dshm\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:06:36.298458 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.298241 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbd638a4-11fa-4a78-ac08-351a0e415f27-tmp-dir\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:06:36.298458 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.298249 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7v7r\" (UniqueName: \"kubernetes.io/projected/dbd638a4-11fa-4a78-ac08-351a0e415f27-kube-api-access-z7v7r\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:06:36.455986 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.455949 2583 generic.go:356] "Generic (PLEG): container finished" podID="dbd638a4-11fa-4a78-ac08-351a0e415f27" containerID="47f96d0a9f5790586bf4a2d82cd9d67821b33aa3cb29948af726121954bc5837" exitCode=0 Sep 11 15:06:36.456208 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.456003 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" Sep 11 15:06:36.456208 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.456047 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" event={"ID":"dbd638a4-11fa-4a78-ac08-351a0e415f27","Type":"ContainerDied","Data":"47f96d0a9f5790586bf4a2d82cd9d67821b33aa3cb29948af726121954bc5837"} Sep 11 15:06:36.456208 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.456118 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7" event={"ID":"dbd638a4-11fa-4a78-ac08-351a0e415f27","Type":"ContainerDied","Data":"ec3648d9c62eed1ff108da7f5012b66f071975085128ea8d6d91d6ad1549ff38"} Sep 11 15:06:36.456208 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.456140 2583 scope.go:117] "RemoveContainer" containerID="47f96d0a9f5790586bf4a2d82cd9d67821b33aa3cb29948af726121954bc5837" Sep 11 15:06:36.464424 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.464390 2583 generic.go:356] "Generic (PLEG): container finished" podID="b3f7c8b2-e74b-4902-a002-104b20fac586" containerID="2529b55b23417c5818b4d99ba52e5e80586767bbbe67a1fe024480725de6f9d2" exitCode=0 Sep 11 15:06:36.464580 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.464561 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" event={"ID":"b3f7c8b2-e74b-4902-a002-104b20fac586","Type":"ContainerDied","Data":"2529b55b23417c5818b4d99ba52e5e80586767bbbe67a1fe024480725de6f9d2"} Sep 11 15:06:36.465204 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.465138 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" event={"ID":"b3f7c8b2-e74b-4902-a002-104b20fac586","Type":"ContainerDied","Data":"3cace6153561e369a70d88345a4bba8d09d6403b097251695eb13c7c05cb2bd0"} Sep 11 15:06:36.466166 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.466132 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk" Sep 11 15:06:36.476899 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.476873 2583 scope.go:117] "RemoveContainer" containerID="47f96d0a9f5790586bf4a2d82cd9d67821b33aa3cb29948af726121954bc5837" Sep 11 15:06:36.477250 ip-10-0-135-39 kubenswrapper[2583]: E0911 15:06:36.477227 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f96d0a9f5790586bf4a2d82cd9d67821b33aa3cb29948af726121954bc5837\": container with ID starting with 47f96d0a9f5790586bf4a2d82cd9d67821b33aa3cb29948af726121954bc5837 not found: ID does not exist" containerID="47f96d0a9f5790586bf4a2d82cd9d67821b33aa3cb29948af726121954bc5837" Sep 11 15:06:36.477313 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.477259 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f96d0a9f5790586bf4a2d82cd9d67821b33aa3cb29948af726121954bc5837"} err="failed to get container status \"47f96d0a9f5790586bf4a2d82cd9d67821b33aa3cb29948af726121954bc5837\": rpc error: code = NotFound desc = could not find container \"47f96d0a9f5790586bf4a2d82cd9d67821b33aa3cb29948af726121954bc5837\": container with ID starting with 47f96d0a9f5790586bf4a2d82cd9d67821b33aa3cb29948af726121954bc5837 not found: ID does not exist" Sep 11 15:06:36.477313 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.477277 2583 scope.go:117] "RemoveContainer" containerID="2529b55b23417c5818b4d99ba52e5e80586767bbbe67a1fe024480725de6f9d2" Sep 11 15:06:36.486031 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.486009 2583 scope.go:117] "RemoveContainer" containerID="2529b55b23417c5818b4d99ba52e5e80586767bbbe67a1fe024480725de6f9d2" Sep 11 15:06:36.486498 ip-10-0-135-39 kubenswrapper[2583]: E0911 15:06:36.486455 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2529b55b23417c5818b4d99ba52e5e80586767bbbe67a1fe024480725de6f9d2\": container with ID starting with 2529b55b23417c5818b4d99ba52e5e80586767bbbe67a1fe024480725de6f9d2 not found: ID does not exist" containerID="2529b55b23417c5818b4d99ba52e5e80586767bbbe67a1fe024480725de6f9d2" Sep 11 15:06:36.486649 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.486506 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2529b55b23417c5818b4d99ba52e5e80586767bbbe67a1fe024480725de6f9d2"} err="failed to get container status \"2529b55b23417c5818b4d99ba52e5e80586767bbbe67a1fe024480725de6f9d2\": rpc error: code = NotFound desc = could not find container \"2529b55b23417c5818b4d99ba52e5e80586767bbbe67a1fe024480725de6f9d2\": container with ID starting with 2529b55b23417c5818b4d99ba52e5e80586767bbbe67a1fe024480725de6f9d2 not found: ID does not exist" Sep 11 15:06:36.489562 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.489538 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7"] Sep 11 15:06:36.493504 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.493481 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-837632/distinct-w4-837632-kserve-865dc95ddb-wmht7"] Sep 11 15:06:36.503733 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.503691 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk"] Sep 11 15:06:36.510539 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:36.510509 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-837632/premium-w4-837632-kserve-865cf6d967-bbqnk"] Sep 11 15:06:37.332233 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:37.332198 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3f7c8b2-e74b-4902-a002-104b20fac586" path="/var/lib/kubelet/pods/b3f7c8b2-e74b-4902-a002-104b20fac586/volumes" Sep 11 15:06:37.332618 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:37.332603 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbd638a4-11fa-4a78-ac08-351a0e415f27" path="/var/lib/kubelet/pods/dbd638a4-11fa-4a78-ac08-351a0e415f27/volumes" Sep 11 15:06:40.211744 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:40.211692 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7"] Sep 11 15:06:40.212243 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:40.211922 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" podUID="9f745a9f-d6ba-4629-b8c0-8faacfbd4845" containerName="istio-proxy" containerID="cri-o://4b3f3873c88ccdaa9cf8949c82ef5165f90a9536d9438b5791668876e75c6e30" gracePeriod=30 Sep 11 15:06:44.203452 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:44.203411 2583 patch_prober.go:28] interesting pod/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 15:06:44.203913 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:44.203485 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" podUID="9f745a9f-d6ba-4629-b8c0-8faacfbd4845" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 15:06:45.470392 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.470363 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 15:06:45.525641 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.525547 2583 generic.go:356] "Generic (PLEG): container finished" podID="9f745a9f-d6ba-4629-b8c0-8faacfbd4845" containerID="4b3f3873c88ccdaa9cf8949c82ef5165f90a9536d9438b5791668876e75c6e30" exitCode=0 Sep 11 15:06:45.525641 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.525603 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" Sep 11 15:06:45.525849 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.525640 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" event={"ID":"9f745a9f-d6ba-4629-b8c0-8faacfbd4845","Type":"ContainerDied","Data":"4b3f3873c88ccdaa9cf8949c82ef5165f90a9536d9438b5791668876e75c6e30"} Sep 11 15:06:45.525849 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.525701 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7" event={"ID":"9f745a9f-d6ba-4629-b8c0-8faacfbd4845","Type":"ContainerDied","Data":"c83fa480ece593bb4a413af4acfd61708f507a5c212331817ade3161034de61e"} Sep 11 15:06:45.525849 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.525742 2583 scope.go:117] "RemoveContainer" containerID="4b3f3873c88ccdaa9cf8949c82ef5165f90a9536d9438b5791668876e75c6e30" Sep 11 15:06:45.535113 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.535094 2583 scope.go:117] "RemoveContainer" containerID="27a4bd6866cdf80cc2ce89ec8dda169627b693d5a2e4e5292cbe4b9c49628f6d" Sep 11 15:06:45.543829 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.543804 2583 scope.go:117] "RemoveContainer" containerID="4b3f3873c88ccdaa9cf8949c82ef5165f90a9536d9438b5791668876e75c6e30" Sep 11 15:06:45.544141 ip-10-0-135-39 kubenswrapper[2583]: E0911 15:06:45.544124 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b3f3873c88ccdaa9cf8949c82ef5165f90a9536d9438b5791668876e75c6e30\": container with ID starting with 4b3f3873c88ccdaa9cf8949c82ef5165f90a9536d9438b5791668876e75c6e30 not found: ID does not exist" containerID="4b3f3873c88ccdaa9cf8949c82ef5165f90a9536d9438b5791668876e75c6e30" Sep 11 15:06:45.544204 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.544150 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b3f3873c88ccdaa9cf8949c82ef5165f90a9536d9438b5791668876e75c6e30"} err="failed to get container status \"4b3f3873c88ccdaa9cf8949c82ef5165f90a9536d9438b5791668876e75c6e30\": rpc error: code = NotFound desc = could not find container \"4b3f3873c88ccdaa9cf8949c82ef5165f90a9536d9438b5791668876e75c6e30\": container with ID starting with 4b3f3873c88ccdaa9cf8949c82ef5165f90a9536d9438b5791668876e75c6e30 not found: ID does not exist" Sep 11 15:06:45.544204 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.544168 2583 scope.go:117] "RemoveContainer" containerID="27a4bd6866cdf80cc2ce89ec8dda169627b693d5a2e4e5292cbe4b9c49628f6d" Sep 11 15:06:45.544443 ip-10-0-135-39 kubenswrapper[2583]: E0911 15:06:45.544424 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27a4bd6866cdf80cc2ce89ec8dda169627b693d5a2e4e5292cbe4b9c49628f6d\": container with ID starting with 27a4bd6866cdf80cc2ce89ec8dda169627b693d5a2e4e5292cbe4b9c49628f6d not found: ID does not exist" containerID="27a4bd6866cdf80cc2ce89ec8dda169627b693d5a2e4e5292cbe4b9c49628f6d" Sep 11 15:06:45.544486 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.544449 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27a4bd6866cdf80cc2ce89ec8dda169627b693d5a2e4e5292cbe4b9c49628f6d"} err="failed to get container status \"27a4bd6866cdf80cc2ce89ec8dda169627b693d5a2e4e5292cbe4b9c49628f6d\": rpc error: code = NotFound desc = could not find container \"27a4bd6866cdf80cc2ce89ec8dda169627b693d5a2e4e5292cbe4b9c49628f6d\": container with ID starting with 27a4bd6866cdf80cc2ce89ec8dda169627b693d5a2e4e5292cbe4b9c49628f6d not found: ID does not exist" Sep 11 15:06:45.583993 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.583953 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txt6c\" (UniqueName: \"kubernetes.io/projected/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-kube-api-access-txt6c\") pod \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " Sep 11 15:06:45.584148 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.584041 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istio-podinfo\") pod \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " Sep 11 15:06:45.584148 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.584086 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istio-token\") pod \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " Sep 11 15:06:45.584148 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.584118 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-credential-socket\") pod \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " Sep 11 15:06:45.584148 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.584141 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-workload-socket\") pod \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " Sep 11 15:06:45.584334 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.584171 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istio-data\") pod \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " Sep 11 15:06:45.584334 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.584196 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-workload-certs\") pod \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " Sep 11 15:06:45.584334 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.584227 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istio-envoy\") pod \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " Sep 11 15:06:45.584334 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.584261 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istiod-ca-cert\") pod \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\" (UID: \"9f745a9f-d6ba-4629-b8c0-8faacfbd4845\") " Sep 11 15:06:45.584538 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.584414 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9f745a9f-d6ba-4629-b8c0-8faacfbd4845" (UID: "9f745a9f-d6ba-4629-b8c0-8faacfbd4845"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:06:45.584538 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.584490 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9f745a9f-d6ba-4629-b8c0-8faacfbd4845" (UID: "9f745a9f-d6ba-4629-b8c0-8faacfbd4845"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:06:45.584538 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.584507 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9f745a9f-d6ba-4629-b8c0-8faacfbd4845" (UID: "9f745a9f-d6ba-4629-b8c0-8faacfbd4845"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:06:45.584745 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.584721 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-credential-socket\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:06:45.584745 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.584748 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-workload-socket\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:06:45.584955 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.584763 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-workload-certs\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:06:45.584955 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.584901 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istio-data" (OuterVolumeSpecName: "istio-data") pod "9f745a9f-d6ba-4629-b8c0-8faacfbd4845" (UID: "9f745a9f-d6ba-4629-b8c0-8faacfbd4845"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:06:45.584955 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.584924 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9f745a9f-d6ba-4629-b8c0-8faacfbd4845" (UID: "9f745a9f-d6ba-4629-b8c0-8faacfbd4845"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:06:45.586258 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.586231 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istio-token" (OuterVolumeSpecName: "istio-token") pod "9f745a9f-d6ba-4629-b8c0-8faacfbd4845" (UID: "9f745a9f-d6ba-4629-b8c0-8faacfbd4845"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:06:45.586359 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.586318 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-kube-api-access-txt6c" (OuterVolumeSpecName: "kube-api-access-txt6c") pod "9f745a9f-d6ba-4629-b8c0-8faacfbd4845" (UID: "9f745a9f-d6ba-4629-b8c0-8faacfbd4845"). InnerVolumeSpecName "kube-api-access-txt6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:06:45.586717 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.586685 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9f745a9f-d6ba-4629-b8c0-8faacfbd4845" (UID: "9f745a9f-d6ba-4629-b8c0-8faacfbd4845"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:06:45.586781 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.586697 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9f745a9f-d6ba-4629-b8c0-8faacfbd4845" (UID: "9f745a9f-d6ba-4629-b8c0-8faacfbd4845"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 15:06:45.686000 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.685960 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istio-podinfo\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:06:45.686000 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.685993 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istio-token\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:06:45.686000 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.686002 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istio-data\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:06:45.686000 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.686010 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istio-envoy\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:06:45.686282 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.686018 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-istiod-ca-cert\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:06:45.686282 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.686026 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txt6c\" (UniqueName: \"kubernetes.io/projected/9f745a9f-d6ba-4629-b8c0-8faacfbd4845-kube-api-access-txt6c\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:06:45.850642 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.850607 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7"] Sep 11 15:06:45.854489 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:45.854458 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-837632-openshift-default-7588449dc8-vh7f7"] Sep 11 15:06:46.695847 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:46.695806 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14a808-jw94p"] Sep 11 15:06:46.699775 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:46.699738 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14a808-jw94p"] Sep 11 15:06:47.330658 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:47.330625 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="498da86c-163a-44ec-a573-0fe185893871" path="/var/lib/kubelet/pods/498da86c-163a-44ec-a573-0fe185893871/volumes" Sep 11 15:06:47.331111 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:47.331096 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f745a9f-d6ba-4629-b8c0-8faacfbd4845" path="/var/lib/kubelet/pods/9f745a9f-d6ba-4629-b8c0-8faacfbd4845/volumes" Sep 11 15:06:54.561792 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:54.561755 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4e39b-nn6qn"] Sep 11 15:06:54.562192 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:54.562179 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f745a9f-d6ba-4629-b8c0-8faacfbd4845" containerName="istio-proxy" Sep 11 15:06:54.562192 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:54.562193 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f745a9f-d6ba-4629-b8c0-8faacfbd4845" containerName="istio-proxy" Sep 11 15:06:54.562275 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:54.562206 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="498da86c-163a-44ec-a573-0fe185893871" containerName="revoke-keys" Sep 11 15:06:54.562275 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:54.562211 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="498da86c-163a-44ec-a573-0fe185893871" containerName="revoke-keys" Sep 11 15:06:54.562275 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:54.562220 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f745a9f-d6ba-4629-b8c0-8faacfbd4845" containerName="istio-proxy" Sep 11 15:06:54.562275 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:54.562225 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f745a9f-d6ba-4629-b8c0-8faacfbd4845" containerName="istio-proxy" Sep 11 15:06:54.562275 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:54.562238 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbd638a4-11fa-4a78-ac08-351a0e415f27" containerName="main" Sep 11 15:06:54.562275 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:54.562243 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbd638a4-11fa-4a78-ac08-351a0e415f27" containerName="main" Sep 11 15:06:54.562275 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:54.562253 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3f7c8b2-e74b-4902-a002-104b20fac586" containerName="main" Sep 11 15:06:54.562275 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:54.562258 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3f7c8b2-e74b-4902-a002-104b20fac586" containerName="main" Sep 11 15:06:54.562529 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:54.562318 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f745a9f-d6ba-4629-b8c0-8faacfbd4845" containerName="istio-proxy" Sep 11 15:06:54.562529 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:54.562325 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="498da86c-163a-44ec-a573-0fe185893871" containerName="revoke-keys" Sep 11 15:06:54.562529 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:54.562334 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3f7c8b2-e74b-4902-a002-104b20fac586" containerName="main" Sep 11 15:06:54.562529 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:54.562340 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f745a9f-d6ba-4629-b8c0-8faacfbd4845" containerName="istio-proxy" Sep 11 15:06:54.562529 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:54.562345 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbd638a4-11fa-4a78-ac08-351a0e415f27" containerName="main" Sep 11 15:06:54.566802 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:54.566784 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4e39b-nn6qn" Sep 11 15:06:54.569373 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:54.569350 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:06:54.569954 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:54.569934 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nks8w\"" Sep 11 15:06:54.577777 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:54.577755 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4e39b-nn6qn"] Sep 11 15:06:54.675720 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:54.675673 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b929aa8f-e0eb-4b08-b121-d96035a293f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c4e39b-nn6qn\" (UID: \"b929aa8f-e0eb-4b08-b121-d96035a293f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4e39b-nn6qn" Sep 11 15:06:54.776567 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:54.776533 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b929aa8f-e0eb-4b08-b121-d96035a293f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c4e39b-nn6qn\" (UID: \"b929aa8f-e0eb-4b08-b121-d96035a293f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4e39b-nn6qn" Sep 11 15:06:54.777190 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:54.777170 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b929aa8f-e0eb-4b08-b121-d96035a293f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c4e39b-nn6qn\" (UID: \"b929aa8f-e0eb-4b08-b121-d96035a293f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4e39b-nn6qn" Sep 11 15:06:54.877619 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:54.877595 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4e39b-nn6qn" Sep 11 15:06:55.150261 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:55.150231 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4e39b-nn6qn"] Sep 11 15:06:55.152143 ip-10-0-135-39 kubenswrapper[2583]: W0911 15:06:55.152111 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb929aa8f_e0eb_4b08_b121_d96035a293f6.slice/crio-7f3947591a9c562eafbafc20eabea9e4cf8c23ff4e705b51e1e88627d4364a49 WatchSource:0}: Error finding container 7f3947591a9c562eafbafc20eabea9e4cf8c23ff4e705b51e1e88627d4364a49: Status 404 returned error can't find the container with id 7f3947591a9c562eafbafc20eabea9e4cf8c23ff4e705b51e1e88627d4364a49 Sep 11 15:06:55.595843 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:55.595745 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4e39b-nn6qn" event={"ID":"b929aa8f-e0eb-4b08-b121-d96035a293f6","Type":"ContainerStarted","Data":"e51fb3fc44ad6bb3e1a29a0776176cf409ef3244b814dfb41cc47c0744adfe09"} Sep 11 15:06:55.595843 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:55.595801 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4e39b-nn6qn" event={"ID":"b929aa8f-e0eb-4b08-b121-d96035a293f6","Type":"ContainerStarted","Data":"7f3947591a9c562eafbafc20eabea9e4cf8c23ff4e705b51e1e88627d4364a49"} Sep 11 15:06:55.614954 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:55.614904 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4e39b-nn6qn" podStartSLOduration=1.614887698 podStartE2EDuration="1.614887698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:06:55.612247378 +0000 UTC m=+1606.856233038" watchObservedRunningTime="2026-09-11 15:06:55.614887698 +0000 UTC m=+1606.858873406" Sep 11 15:06:56.603220 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:56.603183 2583 generic.go:356] "Generic (PLEG): container finished" podID="b929aa8f-e0eb-4b08-b121-d96035a293f6" containerID="e51fb3fc44ad6bb3e1a29a0776176cf409ef3244b814dfb41cc47c0744adfe09" exitCode=0 Sep 11 15:06:56.603615 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:56.603266 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4e39b-nn6qn" event={"ID":"b929aa8f-e0eb-4b08-b121-d96035a293f6","Type":"ContainerDied","Data":"e51fb3fc44ad6bb3e1a29a0776176cf409ef3244b814dfb41cc47c0744adfe09"} Sep 11 15:06:57.747261 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:57.747233 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4e39b-nn6qn" Sep 11 15:06:57.906326 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:57.906295 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b929aa8f-e0eb-4b08-b121-d96035a293f6-maas-api-service-ca\") pod \"b929aa8f-e0eb-4b08-b121-d96035a293f6\" (UID: \"b929aa8f-e0eb-4b08-b121-d96035a293f6\") " Sep 11 15:06:57.906717 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:57.906689 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b929aa8f-e0eb-4b08-b121-d96035a293f6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b929aa8f-e0eb-4b08-b121-d96035a293f6" (UID: "b929aa8f-e0eb-4b08-b121-d96035a293f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:06:58.007618 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:58.007572 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b929aa8f-e0eb-4b08-b121-d96035a293f6-maas-api-service-ca\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:06:58.617164 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:58.617134 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4e39b-nn6qn" Sep 11 15:06:58.617164 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:58.617146 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4e39b-nn6qn" event={"ID":"b929aa8f-e0eb-4b08-b121-d96035a293f6","Type":"ContainerDied","Data":"7f3947591a9c562eafbafc20eabea9e4cf8c23ff4e705b51e1e88627d4364a49"} Sep 11 15:06:58.617378 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:06:58.617196 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f3947591a9c562eafbafc20eabea9e4cf8c23ff4e705b51e1e88627d4364a49" Sep 11 15:07:05.151746 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:05.151528 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c4e39b-29818980-wg887"] Sep 11 15:07:05.154340 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:05.154308 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c4e39b-29818980-wg887"] Sep 11 15:07:05.180307 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:05.180273 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w"] Sep 11 15:07:05.180553 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:05.180527 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w" podUID="8c7e3927-4576-4427-8613-496e7599e40e" containerName="payload-processing" containerID="cri-o://4c901f79ec65d582acbb43e46929a800bec91402f53356d848cc01eb5e936364" gracePeriod=30 Sep 11 15:07:05.332454 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:05.332420 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d476d701-ae6b-40bd-9054-efada8ccd100" path="/var/lib/kubelet/pods/d476d701-ae6b-40bd-9054-efada8ccd100/volumes" Sep 11 15:07:05.375538 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:05.375505 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4e39b-nn6qn"] Sep 11 15:07:05.378309 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:05.378278 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4e39b-nn6qn"] Sep 11 15:07:05.439209 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:05.439181 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w" Sep 11 15:07:05.478019 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:05.477993 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjjj8\" (UniqueName: \"kubernetes.io/projected/8c7e3927-4576-4427-8613-496e7599e40e-kube-api-access-mjjj8\") pod \"8c7e3927-4576-4427-8613-496e7599e40e\" (UID: \"8c7e3927-4576-4427-8613-496e7599e40e\") " Sep 11 15:07:05.478187 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:05.478078 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c7e3927-4576-4427-8613-496e7599e40e-plugins-config-volume\") pod \"8c7e3927-4576-4427-8613-496e7599e40e\" (UID: \"8c7e3927-4576-4427-8613-496e7599e40e\") " Sep 11 15:07:05.478492 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:05.478466 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c7e3927-4576-4427-8613-496e7599e40e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8c7e3927-4576-4427-8613-496e7599e40e" (UID: "8c7e3927-4576-4427-8613-496e7599e40e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:07:05.480535 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:05.480515 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c7e3927-4576-4427-8613-496e7599e40e-kube-api-access-mjjj8" (OuterVolumeSpecName: "kube-api-access-mjjj8") pod "8c7e3927-4576-4427-8613-496e7599e40e" (UID: "8c7e3927-4576-4427-8613-496e7599e40e"). InnerVolumeSpecName "kube-api-access-mjjj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:07:05.578785 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:05.578745 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjjj8\" (UniqueName: \"kubernetes.io/projected/8c7e3927-4576-4427-8613-496e7599e40e-kube-api-access-mjjj8\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:07:05.578785 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:05.578778 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c7e3927-4576-4427-8613-496e7599e40e-plugins-config-volume\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:07:05.661997 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:05.661956 2583 generic.go:356] "Generic (PLEG): container finished" podID="8c7e3927-4576-4427-8613-496e7599e40e" containerID="4c901f79ec65d582acbb43e46929a800bec91402f53356d848cc01eb5e936364" exitCode=0 Sep 11 15:07:05.661997 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:05.661989 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w" event={"ID":"8c7e3927-4576-4427-8613-496e7599e40e","Type":"ContainerDied","Data":"4c901f79ec65d582acbb43e46929a800bec91402f53356d848cc01eb5e936364"} Sep 11 15:07:05.662210 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:05.662033 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w" Sep 11 15:07:05.662210 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:05.662069 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w" event={"ID":"8c7e3927-4576-4427-8613-496e7599e40e","Type":"ContainerDied","Data":"f41ad68eb03de7baa3e8efbeb40bd1822ef9fee6dfdaa0327eca71077538d0a1"} Sep 11 15:07:05.662210 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:05.662092 2583 scope.go:117] "RemoveContainer" containerID="4c901f79ec65d582acbb43e46929a800bec91402f53356d848cc01eb5e936364" Sep 11 15:07:05.673203 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:05.673186 2583 scope.go:117] "RemoveContainer" containerID="4c901f79ec65d582acbb43e46929a800bec91402f53356d848cc01eb5e936364" Sep 11 15:07:05.673501 ip-10-0-135-39 kubenswrapper[2583]: E0911 15:07:05.673479 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c901f79ec65d582acbb43e46929a800bec91402f53356d848cc01eb5e936364\": container with ID starting with 4c901f79ec65d582acbb43e46929a800bec91402f53356d848cc01eb5e936364 not found: ID does not exist" containerID="4c901f79ec65d582acbb43e46929a800bec91402f53356d848cc01eb5e936364" Sep 11 15:07:05.673597 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:05.673512 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c901f79ec65d582acbb43e46929a800bec91402f53356d848cc01eb5e936364"} err="failed to get container status \"4c901f79ec65d582acbb43e46929a800bec91402f53356d848cc01eb5e936364\": rpc error: code = NotFound desc = could not find container \"4c901f79ec65d582acbb43e46929a800bec91402f53356d848cc01eb5e936364\": container with ID starting with 4c901f79ec65d582acbb43e46929a800bec91402f53356d848cc01eb5e936364 not found: ID does not exist" Sep 11 15:07:05.684534 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:05.684510 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w"] Sep 11 15:07:05.693449 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:05.693427 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c4e39b-7c845f5685-8lr4w"] Sep 11 15:07:07.331347 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:07.331303 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c7e3927-4576-4427-8613-496e7599e40e" path="/var/lib/kubelet/pods/8c7e3927-4576-4427-8613-496e7599e40e/volumes" Sep 11 15:07:07.331841 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:07.331808 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b929aa8f-e0eb-4b08-b121-d96035a293f6" path="/var/lib/kubelet/pods/b929aa8f-e0eb-4b08-b121-d96035a293f6/volumes" Sep 11 15:07:09.604008 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:09.603969 2583 scope.go:117] "RemoveContainer" containerID="bb7a24390d0a5460b0bda3ed0a31788544fd9029ae17bfd698eb36355d2da909" Sep 11 15:07:12.709040 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:12.709002 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c0e28e-w8wq7"] Sep 11 15:07:12.709510 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:12.709408 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b929aa8f-e0eb-4b08-b121-d96035a293f6" containerName="revoke-keys" Sep 11 15:07:12.709510 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:12.709421 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b929aa8f-e0eb-4b08-b121-d96035a293f6" containerName="revoke-keys" Sep 11 15:07:12.709510 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:12.709432 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c7e3927-4576-4427-8613-496e7599e40e" containerName="payload-processing" Sep 11 15:07:12.709510 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:12.709438 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c7e3927-4576-4427-8613-496e7599e40e" containerName="payload-processing" Sep 11 15:07:12.709510 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:12.709506 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b929aa8f-e0eb-4b08-b121-d96035a293f6" containerName="revoke-keys" Sep 11 15:07:12.709674 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:12.709517 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c7e3927-4576-4427-8613-496e7599e40e" containerName="payload-processing" Sep 11 15:07:12.712601 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:12.712583 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c0e28e-w8wq7" Sep 11 15:07:12.715091 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:12.715066 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:07:12.715216 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:12.715067 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nks8w\"" Sep 11 15:07:12.722381 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:12.722353 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c0e28e-w8wq7"] Sep 11 15:07:12.748837 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:12.748798 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/234c8195-46e0-4ca4-baad-6a2fb796342b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c0e28e-w8wq7\" (UID: \"234c8195-46e0-4ca4-baad-6a2fb796342b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c0e28e-w8wq7" Sep 11 15:07:12.849689 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:12.849642 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/234c8195-46e0-4ca4-baad-6a2fb796342b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c0e28e-w8wq7\" (UID: \"234c8195-46e0-4ca4-baad-6a2fb796342b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c0e28e-w8wq7" Sep 11 15:07:12.850383 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:12.850354 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/234c8195-46e0-4ca4-baad-6a2fb796342b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c0e28e-w8wq7\" (UID: \"234c8195-46e0-4ca4-baad-6a2fb796342b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c0e28e-w8wq7" Sep 11 15:07:13.024102 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:13.024005 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c0e28e-w8wq7" Sep 11 15:07:13.302370 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:13.302344 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c0e28e-w8wq7"] Sep 11 15:07:13.304239 ip-10-0-135-39 kubenswrapper[2583]: W0911 15:07:13.304212 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod234c8195_46e0_4ca4_baad_6a2fb796342b.slice/crio-ec08bcb292d7b69614eb043ddb0565734f7f00f35ee7962d0ae24b3af2d86bdf WatchSource:0}: Error finding container ec08bcb292d7b69614eb043ddb0565734f7f00f35ee7962d0ae24b3af2d86bdf: Status 404 returned error can't find the container with id ec08bcb292d7b69614eb043ddb0565734f7f00f35ee7962d0ae24b3af2d86bdf Sep 11 15:07:13.713418 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:13.713380 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c0e28e-w8wq7" event={"ID":"234c8195-46e0-4ca4-baad-6a2fb796342b","Type":"ContainerStarted","Data":"6bdd5a0dd05e1c2fed3c7415009e87a2541c72df8331ee603d973f1ff022d697"} Sep 11 15:07:13.713832 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:13.713443 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c0e28e-w8wq7" event={"ID":"234c8195-46e0-4ca4-baad-6a2fb796342b","Type":"ContainerStarted","Data":"ec08bcb292d7b69614eb043ddb0565734f7f00f35ee7962d0ae24b3af2d86bdf"} Sep 11 15:07:13.732607 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:13.732550 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c0e28e-w8wq7" podStartSLOduration=1.732534453 podStartE2EDuration="1.732534453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:07:13.72983453 +0000 UTC m=+1624.973820246" watchObservedRunningTime="2026-09-11 15:07:13.732534453 +0000 UTC m=+1624.976520125" Sep 11 15:07:14.720773 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:14.720738 2583 generic.go:356] "Generic (PLEG): container finished" podID="234c8195-46e0-4ca4-baad-6a2fb796342b" containerID="6bdd5a0dd05e1c2fed3c7415009e87a2541c72df8331ee603d973f1ff022d697" exitCode=0 Sep 11 15:07:14.721158 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:14.720820 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c0e28e-w8wq7" event={"ID":"234c8195-46e0-4ca4-baad-6a2fb796342b","Type":"ContainerDied","Data":"6bdd5a0dd05e1c2fed3c7415009e87a2541c72df8331ee603d973f1ff022d697"} Sep 11 15:07:15.857788 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:15.857759 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c0e28e-w8wq7" Sep 11 15:07:15.981274 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:15.981180 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/234c8195-46e0-4ca4-baad-6a2fb796342b-maas-api-service-ca\") pod \"234c8195-46e0-4ca4-baad-6a2fb796342b\" (UID: \"234c8195-46e0-4ca4-baad-6a2fb796342b\") " Sep 11 15:07:15.981576 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:15.981555 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/234c8195-46e0-4ca4-baad-6a2fb796342b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "234c8195-46e0-4ca4-baad-6a2fb796342b" (UID: "234c8195-46e0-4ca4-baad-6a2fb796342b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:07:16.082140 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:16.082112 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/234c8195-46e0-4ca4-baad-6a2fb796342b-maas-api-service-ca\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:07:16.734482 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:16.734451 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c0e28e-w8wq7" Sep 11 15:07:16.734482 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:16.734465 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c0e28e-w8wq7" event={"ID":"234c8195-46e0-4ca4-baad-6a2fb796342b","Type":"ContainerDied","Data":"ec08bcb292d7b69614eb043ddb0565734f7f00f35ee7962d0ae24b3af2d86bdf"} Sep 11 15:07:16.734787 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:16.734524 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec08bcb292d7b69614eb043ddb0565734f7f00f35ee7962d0ae24b3af2d86bdf" Sep 11 15:07:23.332776 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.332745 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c0e28e-29818980-nxpmt"] Sep 11 15:07:23.335038 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.335014 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c0e28e-29818980-nxpmt"] Sep 11 15:07:23.361194 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.361164 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt"] Sep 11 15:07:23.361395 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.361371 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt" podUID="b4772d28-a1c5-454b-ace5-903df84a4728" containerName="payload-processing" containerID="cri-o://753f61ca23b0ba4bf741fce42e96e868218abaef4409ac53e5100f34f7d8f265" gracePeriod=30 Sep 11 15:07:23.370938 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.370908 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl"] Sep 11 15:07:23.371203 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.371136 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl" podUID="7f6eb126-f22c-4ca2-a056-5e160c0fc3cf" containerName="payload-pre-processing" containerID="cri-o://a3f56e266b1138defd0af1eb78e946c930d736311995f4fdd1d318f08a38e094" gracePeriod=30 Sep 11 15:07:23.573629 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.573546 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c0e28e-w8wq7"] Sep 11 15:07:23.580734 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.580679 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c0e28e-w8wq7"] Sep 11 15:07:23.656384 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.656358 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt" Sep 11 15:07:23.659237 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.659216 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl" Sep 11 15:07:23.753881 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.753842 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f6eb126-f22c-4ca2-a056-5e160c0fc3cf-plugins-config-volume\") pod \"7f6eb126-f22c-4ca2-a056-5e160c0fc3cf\" (UID: \"7f6eb126-f22c-4ca2-a056-5e160c0fc3cf\") " Sep 11 15:07:23.754077 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.753900 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xc855\" (UniqueName: \"kubernetes.io/projected/b4772d28-a1c5-454b-ace5-903df84a4728-kube-api-access-xc855\") pod \"b4772d28-a1c5-454b-ace5-903df84a4728\" (UID: \"b4772d28-a1c5-454b-ace5-903df84a4728\") " Sep 11 15:07:23.754077 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.753969 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4772d28-a1c5-454b-ace5-903df84a4728-plugins-config-volume\") pod \"b4772d28-a1c5-454b-ace5-903df84a4728\" (UID: \"b4772d28-a1c5-454b-ace5-903df84a4728\") " Sep 11 15:07:23.754210 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.754129 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f624r\" (UniqueName: \"kubernetes.io/projected/7f6eb126-f22c-4ca2-a056-5e160c0fc3cf-kube-api-access-f624r\") pod \"7f6eb126-f22c-4ca2-a056-5e160c0fc3cf\" (UID: \"7f6eb126-f22c-4ca2-a056-5e160c0fc3cf\") " Sep 11 15:07:23.754210 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.754190 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f6eb126-f22c-4ca2-a056-5e160c0fc3cf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7f6eb126-f22c-4ca2-a056-5e160c0fc3cf" (UID: "7f6eb126-f22c-4ca2-a056-5e160c0fc3cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:07:23.754353 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.754332 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4772d28-a1c5-454b-ace5-903df84a4728-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b4772d28-a1c5-454b-ace5-903df84a4728" (UID: "b4772d28-a1c5-454b-ace5-903df84a4728"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:07:23.754463 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.754439 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4772d28-a1c5-454b-ace5-903df84a4728-plugins-config-volume\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:07:23.754463 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.754455 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f6eb126-f22c-4ca2-a056-5e160c0fc3cf-plugins-config-volume\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:07:23.756129 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.756103 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4772d28-a1c5-454b-ace5-903df84a4728-kube-api-access-xc855" (OuterVolumeSpecName: "kube-api-access-xc855") pod "b4772d28-a1c5-454b-ace5-903df84a4728" (UID: "b4772d28-a1c5-454b-ace5-903df84a4728"). InnerVolumeSpecName "kube-api-access-xc855". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:07:23.756224 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.756194 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f6eb126-f22c-4ca2-a056-5e160c0fc3cf-kube-api-access-f624r" (OuterVolumeSpecName: "kube-api-access-f624r") pod "7f6eb126-f22c-4ca2-a056-5e160c0fc3cf" (UID: "7f6eb126-f22c-4ca2-a056-5e160c0fc3cf"). InnerVolumeSpecName "kube-api-access-f624r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:07:23.776164 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.776129 2583 generic.go:356] "Generic (PLEG): container finished" podID="7f6eb126-f22c-4ca2-a056-5e160c0fc3cf" containerID="a3f56e266b1138defd0af1eb78e946c930d736311995f4fdd1d318f08a38e094" exitCode=0 Sep 11 15:07:23.776362 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.776198 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl" Sep 11 15:07:23.776362 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.776209 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl" event={"ID":"7f6eb126-f22c-4ca2-a056-5e160c0fc3cf","Type":"ContainerDied","Data":"a3f56e266b1138defd0af1eb78e946c930d736311995f4fdd1d318f08a38e094"} Sep 11 15:07:23.776362 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.776299 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl" event={"ID":"7f6eb126-f22c-4ca2-a056-5e160c0fc3cf","Type":"ContainerDied","Data":"b82dd710af0da597d7da5033c4d474ad62ee02c8d4cada093df6f4c8b0427df7"} Sep 11 15:07:23.776362 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.776328 2583 scope.go:117] "RemoveContainer" containerID="a3f56e266b1138defd0af1eb78e946c930d736311995f4fdd1d318f08a38e094" Sep 11 15:07:23.777580 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.777560 2583 generic.go:356] "Generic (PLEG): container finished" podID="b4772d28-a1c5-454b-ace5-903df84a4728" containerID="753f61ca23b0ba4bf741fce42e96e868218abaef4409ac53e5100f34f7d8f265" exitCode=0 Sep 11 15:07:23.777659 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.777603 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt" Sep 11 15:07:23.777659 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.777638 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt" event={"ID":"b4772d28-a1c5-454b-ace5-903df84a4728","Type":"ContainerDied","Data":"753f61ca23b0ba4bf741fce42e96e868218abaef4409ac53e5100f34f7d8f265"} Sep 11 15:07:23.777804 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.777681 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt" event={"ID":"b4772d28-a1c5-454b-ace5-903df84a4728","Type":"ContainerDied","Data":"012053e3ce886e1dad535165853368ea651d6087ba9dca02807b754b49c36184"} Sep 11 15:07:23.786300 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.786275 2583 scope.go:117] "RemoveContainer" containerID="a3f56e266b1138defd0af1eb78e946c930d736311995f4fdd1d318f08a38e094" Sep 11 15:07:23.786676 ip-10-0-135-39 kubenswrapper[2583]: E0911 15:07:23.786649 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3f56e266b1138defd0af1eb78e946c930d736311995f4fdd1d318f08a38e094\": container with ID starting with a3f56e266b1138defd0af1eb78e946c930d736311995f4fdd1d318f08a38e094 not found: ID does not exist" containerID="a3f56e266b1138defd0af1eb78e946c930d736311995f4fdd1d318f08a38e094" Sep 11 15:07:23.786803 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.786687 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3f56e266b1138defd0af1eb78e946c930d736311995f4fdd1d318f08a38e094"} err="failed to get container status \"a3f56e266b1138defd0af1eb78e946c930d736311995f4fdd1d318f08a38e094\": rpc error: code = NotFound desc = could not find container \"a3f56e266b1138defd0af1eb78e946c930d736311995f4fdd1d318f08a38e094\": container with ID starting with a3f56e266b1138defd0af1eb78e946c930d736311995f4fdd1d318f08a38e094 not found: ID does not exist" Sep 11 15:07:23.786803 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.786726 2583 scope.go:117] "RemoveContainer" containerID="753f61ca23b0ba4bf741fce42e96e868218abaef4409ac53e5100f34f7d8f265" Sep 11 15:07:23.795724 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.795687 2583 scope.go:117] "RemoveContainer" containerID="753f61ca23b0ba4bf741fce42e96e868218abaef4409ac53e5100f34f7d8f265" Sep 11 15:07:23.796064 ip-10-0-135-39 kubenswrapper[2583]: E0911 15:07:23.796040 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"753f61ca23b0ba4bf741fce42e96e868218abaef4409ac53e5100f34f7d8f265\": container with ID starting with 753f61ca23b0ba4bf741fce42e96e868218abaef4409ac53e5100f34f7d8f265 not found: ID does not exist" containerID="753f61ca23b0ba4bf741fce42e96e868218abaef4409ac53e5100f34f7d8f265" Sep 11 15:07:23.796129 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.796078 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"753f61ca23b0ba4bf741fce42e96e868218abaef4409ac53e5100f34f7d8f265"} err="failed to get container status \"753f61ca23b0ba4bf741fce42e96e868218abaef4409ac53e5100f34f7d8f265\": rpc error: code = NotFound desc = could not find container \"753f61ca23b0ba4bf741fce42e96e868218abaef4409ac53e5100f34f7d8f265\": container with ID starting with 753f61ca23b0ba4bf741fce42e96e868218abaef4409ac53e5100f34f7d8f265 not found: ID does not exist" Sep 11 15:07:23.807187 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.807155 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt"] Sep 11 15:07:23.809566 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.809545 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c0e28e-f4c8fc48f-krfwt"] Sep 11 15:07:23.820088 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.820063 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl"] Sep 11 15:07:23.824403 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.824380 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c0e28e-7b995c7458-zrvkl"] Sep 11 15:07:23.855494 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.855415 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc855\" (UniqueName: \"kubernetes.io/projected/b4772d28-a1c5-454b-ace5-903df84a4728-kube-api-access-xc855\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:07:23.855494 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:23.855443 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f624r\" (UniqueName: \"kubernetes.io/projected/7f6eb126-f22c-4ca2-a056-5e160c0fc3cf-kube-api-access-f624r\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:07:25.330674 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:25.330641 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="234c8195-46e0-4ca4-baad-6a2fb796342b" path="/var/lib/kubelet/pods/234c8195-46e0-4ca4-baad-6a2fb796342b/volumes" Sep 11 15:07:25.331070 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:25.330979 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fd5471a-377b-4400-b1c4-eedb771ac1b4" path="/var/lib/kubelet/pods/6fd5471a-377b-4400-b1c4-eedb771ac1b4/volumes" Sep 11 15:07:25.331273 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:25.331259 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f6eb126-f22c-4ca2-a056-5e160c0fc3cf" path="/var/lib/kubelet/pods/7f6eb126-f22c-4ca2-a056-5e160c0fc3cf/volumes" Sep 11 15:07:25.331576 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:07:25.331565 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4772d28-a1c5-454b-ace5-903df84a4728" path="/var/lib/kubelet/pods/b4772d28-a1c5-454b-ace5-903df84a4728/volumes" Sep 11 15:08:09.652516 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:08:09.652478 2583 scope.go:117] "RemoveContainer" containerID="7646db0c1de38dc73076aa505b84ef7053f2701070783e1b8e263afeafa57a7d" Sep 11 15:09:09.683201 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:09:09.683167 2583 scope.go:117] "RemoveContainer" containerID="cd2e2b209f5f9e732ad4ff9442e7febcaa4d4692194dc752a81d23f73ee0f2df" Sep 11 15:10:09.344103 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:09.344071 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lwrt2_57d4ba26-384d-42f8-86a1-335e27cdf762/console-operator/1.log" Sep 11 15:10:09.349167 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:09.349140 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lwrt2_57d4ba26-384d-42f8-86a1-335e27cdf762/console-operator/1.log" Sep 11 15:10:16.953315 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:16.953271 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww"] Sep 11 15:10:16.953731 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:16.953647 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f6eb126-f22c-4ca2-a056-5e160c0fc3cf" containerName="payload-pre-processing" Sep 11 15:10:16.953731 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:16.953658 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f6eb126-f22c-4ca2-a056-5e160c0fc3cf" containerName="payload-pre-processing" Sep 11 15:10:16.953731 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:16.953671 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="234c8195-46e0-4ca4-baad-6a2fb796342b" containerName="revoke-keys" Sep 11 15:10:16.953731 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:16.953677 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="234c8195-46e0-4ca4-baad-6a2fb796342b" containerName="revoke-keys" Sep 11 15:10:16.953731 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:16.953683 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4772d28-a1c5-454b-ace5-903df84a4728" containerName="payload-processing" Sep 11 15:10:16.953731 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:16.953689 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4772d28-a1c5-454b-ace5-903df84a4728" containerName="payload-processing" Sep 11 15:10:16.953969 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:16.953786 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="234c8195-46e0-4ca4-baad-6a2fb796342b" containerName="revoke-keys" Sep 11 15:10:16.953969 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:16.953797 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4772d28-a1c5-454b-ace5-903df84a4728" containerName="payload-processing" Sep 11 15:10:16.953969 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:16.953804 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f6eb126-f22c-4ca2-a056-5e160c0fc3cf" containerName="payload-pre-processing" Sep 11 15:10:16.956752 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:16.956730 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww" Sep 11 15:10:16.959392 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:16.959368 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-177c81\"" Sep 11 15:10:16.959522 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:16.959473 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-177c81-dockercfg-lhgdl\"" Sep 11 15:10:16.967003 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:16.966973 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww"] Sep 11 15:10:17.084243 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:17.084207 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dqsc\" (UniqueName: \"kubernetes.io/projected/7c93ac22-6e88-4d5c-9afc-60cc97600e15-kube-api-access-9dqsc\") pod \"payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww\" (UID: \"7c93ac22-6e88-4d5c-9afc-60cc97600e15\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww" Sep 11 15:10:17.084428 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:17.084267 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c93ac22-6e88-4d5c-9afc-60cc97600e15-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww\" (UID: \"7c93ac22-6e88-4d5c-9afc-60cc97600e15\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww" Sep 11 15:10:17.185464 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:17.185433 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dqsc\" (UniqueName: \"kubernetes.io/projected/7c93ac22-6e88-4d5c-9afc-60cc97600e15-kube-api-access-9dqsc\") pod \"payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww\" (UID: \"7c93ac22-6e88-4d5c-9afc-60cc97600e15\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww" Sep 11 15:10:17.185660 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:17.185480 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c93ac22-6e88-4d5c-9afc-60cc97600e15-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww\" (UID: \"7c93ac22-6e88-4d5c-9afc-60cc97600e15\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww" Sep 11 15:10:17.186146 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:17.186127 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c93ac22-6e88-4d5c-9afc-60cc97600e15-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww\" (UID: \"7c93ac22-6e88-4d5c-9afc-60cc97600e15\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww" Sep 11 15:10:17.194487 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:17.194466 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dqsc\" (UniqueName: \"kubernetes.io/projected/7c93ac22-6e88-4d5c-9afc-60cc97600e15-kube-api-access-9dqsc\") pod \"payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww\" (UID: \"7c93ac22-6e88-4d5c-9afc-60cc97600e15\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww" Sep 11 15:10:17.268562 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:17.268477 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww" Sep 11 15:10:17.545470 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:17.545321 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww"] Sep 11 15:10:17.548096 ip-10-0-135-39 kubenswrapper[2583]: W0911 15:10:17.548064 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c93ac22_6e88_4d5c_9afc_60cc97600e15.slice/crio-1977170e136dec4a4d0bd1883f327a5b47a15349f089433aea4a0a851b94b6ad WatchSource:0}: Error finding container 1977170e136dec4a4d0bd1883f327a5b47a15349f089433aea4a0a851b94b6ad: Status 404 returned error can't find the container with id 1977170e136dec4a4d0bd1883f327a5b47a15349f089433aea4a0a851b94b6ad Sep 11 15:10:17.768671 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:17.768629 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww" event={"ID":"7c93ac22-6e88-4d5c-9afc-60cc97600e15","Type":"ContainerStarted","Data":"aa4aba040fb67f35e38747d31aa273c451329ee030bf523a1fd19fa5bf18117e"} Sep 11 15:10:17.768874 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:17.768697 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww" event={"ID":"7c93ac22-6e88-4d5c-9afc-60cc97600e15","Type":"ContainerStarted","Data":"1977170e136dec4a4d0bd1883f327a5b47a15349f089433aea4a0a851b94b6ad"} Sep 11 15:10:17.768874 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:17.768816 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww" Sep 11 15:10:17.788624 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:17.788574 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww" podStartSLOduration=1.788558788 podStartE2EDuration="1.788558788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:10:17.786609449 +0000 UTC m=+1809.030595113" watchObservedRunningTime="2026-09-11 15:10:17.788558788 +0000 UTC m=+1809.032544447" Sep 11 15:10:28.779789 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:28.779756 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww" Sep 11 15:10:46.733462 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:46.733430 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf"] Sep 11 15:10:46.736873 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:46.736855 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf" Sep 11 15:10:46.739377 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:46.739353 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4ee29a\"" Sep 11 15:10:46.739910 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:46.739890 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4ee29a-dockercfg-hfb78\"" Sep 11 15:10:46.748051 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:46.748031 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf"] Sep 11 15:10:46.849847 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:46.849817 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6507a31-c6b7-46f1-b022-6f54b72b1807-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf\" (UID: \"b6507a31-c6b7-46f1-b022-6f54b72b1807\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf" Sep 11 15:10:46.850017 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:46.849893 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvr4k\" (UniqueName: \"kubernetes.io/projected/b6507a31-c6b7-46f1-b022-6f54b72b1807-kube-api-access-dvr4k\") pod \"payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf\" (UID: \"b6507a31-c6b7-46f1-b022-6f54b72b1807\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf" Sep 11 15:10:46.951345 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:46.951319 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6507a31-c6b7-46f1-b022-6f54b72b1807-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf\" (UID: \"b6507a31-c6b7-46f1-b022-6f54b72b1807\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf" Sep 11 15:10:46.951500 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:46.951355 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvr4k\" (UniqueName: \"kubernetes.io/projected/b6507a31-c6b7-46f1-b022-6f54b72b1807-kube-api-access-dvr4k\") pod \"payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf\" (UID: \"b6507a31-c6b7-46f1-b022-6f54b72b1807\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf" Sep 11 15:10:46.952022 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:46.952003 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6507a31-c6b7-46f1-b022-6f54b72b1807-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf\" (UID: \"b6507a31-c6b7-46f1-b022-6f54b72b1807\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf" Sep 11 15:10:46.959598 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:46.959571 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvr4k\" (UniqueName: \"kubernetes.io/projected/b6507a31-c6b7-46f1-b022-6f54b72b1807-kube-api-access-dvr4k\") pod \"payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf\" (UID: \"b6507a31-c6b7-46f1-b022-6f54b72b1807\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf" Sep 11 15:10:47.048858 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:47.048781 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf" Sep 11 15:10:47.322629 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:47.322588 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf"] Sep 11 15:10:47.324254 ip-10-0-135-39 kubenswrapper[2583]: W0911 15:10:47.324225 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6507a31_c6b7_46f1_b022_6f54b72b1807.slice/crio-608a5e168ae0781b06ac6c5a136168009e6257728ba55a8e14acfcd30d04778c WatchSource:0}: Error finding container 608a5e168ae0781b06ac6c5a136168009e6257728ba55a8e14acfcd30d04778c: Status 404 returned error can't find the container with id 608a5e168ae0781b06ac6c5a136168009e6257728ba55a8e14acfcd30d04778c Sep 11 15:10:47.953331 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:47.953297 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf" event={"ID":"b6507a31-c6b7-46f1-b022-6f54b72b1807","Type":"ContainerStarted","Data":"6ca6433e5c7a9c2fb0a9d7eb28295326653c0ee9486fd104abc2dd7813611d13"} Sep 11 15:10:47.953818 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:47.953352 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf" event={"ID":"b6507a31-c6b7-46f1-b022-6f54b72b1807","Type":"ContainerStarted","Data":"608a5e168ae0781b06ac6c5a136168009e6257728ba55a8e14acfcd30d04778c"} Sep 11 15:10:47.953818 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:47.953510 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf" Sep 11 15:10:47.972852 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:47.972789 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf" podStartSLOduration=1.972765367 podStartE2EDuration="1.972765367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:10:47.970965469 +0000 UTC m=+1839.214951129" watchObservedRunningTime="2026-09-11 15:10:47.972765367 +0000 UTC m=+1839.216751028" Sep 11 15:10:58.961478 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:10:58.961447 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf" Sep 11 15:11:09.715481 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:11:09.715451 2583 scope.go:117] "RemoveContainer" containerID="79dc2b93868b841f4a942a2a0a6c315900feed34b2aae0693d68aa94b739a8ef" Sep 11 15:11:13.785029 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:11:13.784993 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf_b6507a31-c6b7-46f1-b022-6f54b72b1807/payload-processing/0.log" Sep 11 15:12:05.219637 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:05.219605 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww_7c93ac22-6e88-4d5c-9afc-60cc97600e15/payload-processing/0.log" Sep 11 15:12:05.487947 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:05.487866 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf_b6507a31-c6b7-46f1-b022-6f54b72b1807/payload-processing/0.log" Sep 11 15:12:09.735350 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:09.735320 2583 scope.go:117] "RemoveContainer" containerID="5fe9812e427d6b2eacc9c7dd0272497eebdd3649b807060ef75e2889d74167ba" Sep 11 15:12:13.643057 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:13.643016 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc"] Sep 11 15:12:13.646590 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:13.646567 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc" Sep 11 15:12:13.648951 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:13.648925 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-e04dee\"" Sep 11 15:12:13.649082 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:13.648951 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-e04dee-dockercfg-q24cg\"" Sep 11 15:12:13.657859 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:13.657823 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc"] Sep 11 15:12:13.699768 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:13.699734 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11ce60c2-ca2f-49c7-aba4-2ac0fff8f217-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc\" (UID: \"11ce60c2-ca2f-49c7-aba4-2ac0fff8f217\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc" Sep 11 15:12:13.699898 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:13.699772 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48wgw\" (UniqueName: \"kubernetes.io/projected/11ce60c2-ca2f-49c7-aba4-2ac0fff8f217-kube-api-access-48wgw\") pod \"payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc\" (UID: \"11ce60c2-ca2f-49c7-aba4-2ac0fff8f217\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc" Sep 11 15:12:13.800624 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:13.800580 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11ce60c2-ca2f-49c7-aba4-2ac0fff8f217-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc\" (UID: \"11ce60c2-ca2f-49c7-aba4-2ac0fff8f217\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc" Sep 11 15:12:13.800847 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:13.800658 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48wgw\" (UniqueName: \"kubernetes.io/projected/11ce60c2-ca2f-49c7-aba4-2ac0fff8f217-kube-api-access-48wgw\") pod \"payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc\" (UID: \"11ce60c2-ca2f-49c7-aba4-2ac0fff8f217\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc" Sep 11 15:12:13.801412 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:13.801385 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11ce60c2-ca2f-49c7-aba4-2ac0fff8f217-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc\" (UID: \"11ce60c2-ca2f-49c7-aba4-2ac0fff8f217\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc" Sep 11 15:12:13.808657 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:13.808634 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48wgw\" (UniqueName: \"kubernetes.io/projected/11ce60c2-ca2f-49c7-aba4-2ac0fff8f217-kube-api-access-48wgw\") pod \"payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc\" (UID: \"11ce60c2-ca2f-49c7-aba4-2ac0fff8f217\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc" Sep 11 15:12:13.957677 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:13.957608 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc" Sep 11 15:12:14.230884 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:14.230855 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc"] Sep 11 15:12:14.232378 ip-10-0-135-39 kubenswrapper[2583]: W0911 15:12:14.232345 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ce60c2_ca2f_49c7_aba4_2ac0fff8f217.slice/crio-14edf6ade61b14fe606741f832982930ffd018a2f1146f525967a8d30a5b5144 WatchSource:0}: Error finding container 14edf6ade61b14fe606741f832982930ffd018a2f1146f525967a8d30a5b5144: Status 404 returned error can't find the container with id 14edf6ade61b14fe606741f832982930ffd018a2f1146f525967a8d30a5b5144 Sep 11 15:12:14.445570 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:14.445532 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc" event={"ID":"11ce60c2-ca2f-49c7-aba4-2ac0fff8f217","Type":"ContainerStarted","Data":"14d84c3658cd9fd34862381d952aa2f2e26ac7a3bf2f18092dbdc0c057ed2da6"} Sep 11 15:12:14.445861 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:14.445833 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc" Sep 11 15:12:14.445970 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:14.445873 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc" event={"ID":"11ce60c2-ca2f-49c7-aba4-2ac0fff8f217","Type":"ContainerStarted","Data":"14edf6ade61b14fe606741f832982930ffd018a2f1146f525967a8d30a5b5144"} Sep 11 15:12:14.465046 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:14.465000 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc" podStartSLOduration=1.4649824790000001 podStartE2EDuration="1.464982479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:12:14.464380498 +0000 UTC m=+1925.708366158" watchObservedRunningTime="2026-09-11 15:12:14.464982479 +0000 UTC m=+1925.708968141" Sep 11 15:12:25.452865 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:25.452837 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc" Sep 11 15:12:39.906534 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:39.906501 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e04dee-qjbhc"] Sep 11 15:12:39.910123 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:39.910106 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e04dee-qjbhc" Sep 11 15:12:39.912408 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:39.912390 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nks8w\"" Sep 11 15:12:39.912493 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:39.912395 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:12:39.919022 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:39.918994 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e04dee-qjbhc"] Sep 11 15:12:39.932963 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:39.932940 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b95590c0-2f23-49ba-9d96-d18451d8b73c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e04dee-qjbhc\" (UID: \"b95590c0-2f23-49ba-9d96-d18451d8b73c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e04dee-qjbhc" Sep 11 15:12:40.034230 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:40.034199 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b95590c0-2f23-49ba-9d96-d18451d8b73c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e04dee-qjbhc\" (UID: \"b95590c0-2f23-49ba-9d96-d18451d8b73c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e04dee-qjbhc" Sep 11 15:12:40.034972 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:40.034948 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b95590c0-2f23-49ba-9d96-d18451d8b73c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e04dee-qjbhc\" (UID: \"b95590c0-2f23-49ba-9d96-d18451d8b73c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e04dee-qjbhc" Sep 11 15:12:40.221516 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:40.221449 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e04dee-qjbhc" Sep 11 15:12:40.494624 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:40.494593 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e04dee-qjbhc"] Sep 11 15:12:40.496568 ip-10-0-135-39 kubenswrapper[2583]: W0911 15:12:40.496542 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb95590c0_2f23_49ba_9d96_d18451d8b73c.slice/crio-097c012a460e26df072ef521f66746f932eddefb3da3475d6fef8f765ecf296d WatchSource:0}: Error finding container 097c012a460e26df072ef521f66746f932eddefb3da3475d6fef8f765ecf296d: Status 404 returned error can't find the container with id 097c012a460e26df072ef521f66746f932eddefb3da3475d6fef8f765ecf296d Sep 11 15:12:40.594929 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:40.594892 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e04dee-qjbhc" event={"ID":"b95590c0-2f23-49ba-9d96-d18451d8b73c","Type":"ContainerStarted","Data":"097c012a460e26df072ef521f66746f932eddefb3da3475d6fef8f765ecf296d"} Sep 11 15:12:41.602006 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:41.601915 2583 generic.go:356] "Generic (PLEG): container finished" podID="b95590c0-2f23-49ba-9d96-d18451d8b73c" containerID="0612f85964d9dc7c453f69735ac4b2ab1687cd74c769f85e3ae400f9a3c75e18" exitCode=0 Sep 11 15:12:41.602006 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:41.601962 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e04dee-qjbhc" event={"ID":"b95590c0-2f23-49ba-9d96-d18451d8b73c","Type":"ContainerDied","Data":"0612f85964d9dc7c453f69735ac4b2ab1687cd74c769f85e3ae400f9a3c75e18"} Sep 11 15:12:42.743196 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:42.743169 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e04dee-qjbhc" Sep 11 15:12:42.756633 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:42.756610 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b95590c0-2f23-49ba-9d96-d18451d8b73c-maas-api-service-ca\") pod \"b95590c0-2f23-49ba-9d96-d18451d8b73c\" (UID: \"b95590c0-2f23-49ba-9d96-d18451d8b73c\") " Sep 11 15:12:42.756968 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:42.756945 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b95590c0-2f23-49ba-9d96-d18451d8b73c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b95590c0-2f23-49ba-9d96-d18451d8b73c" (UID: "b95590c0-2f23-49ba-9d96-d18451d8b73c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:12:42.858114 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:42.858087 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b95590c0-2f23-49ba-9d96-d18451d8b73c-maas-api-service-ca\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:12:43.615690 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:43.615613 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e04dee-qjbhc" Sep 11 15:12:43.615690 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:43.615653 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e04dee-qjbhc" event={"ID":"b95590c0-2f23-49ba-9d96-d18451d8b73c","Type":"ContainerDied","Data":"097c012a460e26df072ef521f66746f932eddefb3da3475d6fef8f765ecf296d"} Sep 11 15:12:43.615897 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:43.615698 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="097c012a460e26df072ef521f66746f932eddefb3da3475d6fef8f765ecf296d" Sep 11 15:12:50.852970 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:50.852936 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc"] Sep 11 15:12:50.853497 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:50.853178 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc" podUID="11ce60c2-ca2f-49c7-aba4-2ac0fff8f217" containerName="payload-pre-processing" containerID="cri-o://14d84c3658cd9fd34862381d952aa2f2e26ac7a3bf2f18092dbdc0c057ed2da6" gracePeriod=30 Sep 11 15:12:51.041746 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:51.041679 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e04dee-qjbhc"] Sep 11 15:12:51.045051 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:51.045023 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e04dee-qjbhc"] Sep 11 15:12:51.093385 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:51.093362 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc" Sep 11 15:12:51.128321 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:51.128296 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11ce60c2-ca2f-49c7-aba4-2ac0fff8f217-plugins-config-volume\") pod \"11ce60c2-ca2f-49c7-aba4-2ac0fff8f217\" (UID: \"11ce60c2-ca2f-49c7-aba4-2ac0fff8f217\") " Sep 11 15:12:51.128501 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:51.128417 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48wgw\" (UniqueName: \"kubernetes.io/projected/11ce60c2-ca2f-49c7-aba4-2ac0fff8f217-kube-api-access-48wgw\") pod \"11ce60c2-ca2f-49c7-aba4-2ac0fff8f217\" (UID: \"11ce60c2-ca2f-49c7-aba4-2ac0fff8f217\") " Sep 11 15:12:51.128772 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:51.128745 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ce60c2-ca2f-49c7-aba4-2ac0fff8f217-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "11ce60c2-ca2f-49c7-aba4-2ac0fff8f217" (UID: "11ce60c2-ca2f-49c7-aba4-2ac0fff8f217"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:12:51.130568 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:51.130517 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ce60c2-ca2f-49c7-aba4-2ac0fff8f217-kube-api-access-48wgw" (OuterVolumeSpecName: "kube-api-access-48wgw") pod "11ce60c2-ca2f-49c7-aba4-2ac0fff8f217" (UID: "11ce60c2-ca2f-49c7-aba4-2ac0fff8f217"). InnerVolumeSpecName "kube-api-access-48wgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:12:51.229427 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:51.229397 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48wgw\" (UniqueName: \"kubernetes.io/projected/11ce60c2-ca2f-49c7-aba4-2ac0fff8f217-kube-api-access-48wgw\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:12:51.229427 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:51.229424 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11ce60c2-ca2f-49c7-aba4-2ac0fff8f217-plugins-config-volume\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:12:51.330621 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:51.330585 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b95590c0-2f23-49ba-9d96-d18451d8b73c" path="/var/lib/kubelet/pods/b95590c0-2f23-49ba-9d96-d18451d8b73c/volumes" Sep 11 15:12:51.664624 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:51.664593 2583 generic.go:356] "Generic (PLEG): container finished" podID="11ce60c2-ca2f-49c7-aba4-2ac0fff8f217" containerID="14d84c3658cd9fd34862381d952aa2f2e26ac7a3bf2f18092dbdc0c057ed2da6" exitCode=0 Sep 11 15:12:51.664809 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:51.664637 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc" Sep 11 15:12:51.664809 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:51.664677 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc" event={"ID":"11ce60c2-ca2f-49c7-aba4-2ac0fff8f217","Type":"ContainerDied","Data":"14d84c3658cd9fd34862381d952aa2f2e26ac7a3bf2f18092dbdc0c057ed2da6"} Sep 11 15:12:51.664809 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:51.664753 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc" event={"ID":"11ce60c2-ca2f-49c7-aba4-2ac0fff8f217","Type":"ContainerDied","Data":"14edf6ade61b14fe606741f832982930ffd018a2f1146f525967a8d30a5b5144"} Sep 11 15:12:51.664809 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:51.664781 2583 scope.go:117] "RemoveContainer" containerID="14d84c3658cd9fd34862381d952aa2f2e26ac7a3bf2f18092dbdc0c057ed2da6" Sep 11 15:12:51.673933 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:51.673905 2583 scope.go:117] "RemoveContainer" containerID="14d84c3658cd9fd34862381d952aa2f2e26ac7a3bf2f18092dbdc0c057ed2da6" Sep 11 15:12:51.674214 ip-10-0-135-39 kubenswrapper[2583]: E0911 15:12:51.674197 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14d84c3658cd9fd34862381d952aa2f2e26ac7a3bf2f18092dbdc0c057ed2da6\": container with ID starting with 14d84c3658cd9fd34862381d952aa2f2e26ac7a3bf2f18092dbdc0c057ed2da6 not found: ID does not exist" containerID="14d84c3658cd9fd34862381d952aa2f2e26ac7a3bf2f18092dbdc0c057ed2da6" Sep 11 15:12:51.674282 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:51.674221 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14d84c3658cd9fd34862381d952aa2f2e26ac7a3bf2f18092dbdc0c057ed2da6"} err="failed to get container status \"14d84c3658cd9fd34862381d952aa2f2e26ac7a3bf2f18092dbdc0c057ed2da6\": rpc error: code = NotFound desc = could not find container \"14d84c3658cd9fd34862381d952aa2f2e26ac7a3bf2f18092dbdc0c057ed2da6\": container with ID starting with 14d84c3658cd9fd34862381d952aa2f2e26ac7a3bf2f18092dbdc0c057ed2da6 not found: ID does not exist" Sep 11 15:12:51.682047 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:51.682023 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc"] Sep 11 15:12:51.688091 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:51.688070 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e04dee-7b7d4886b-2srmc"] Sep 11 15:12:53.330938 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:12:53.330901 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11ce60c2-ca2f-49c7-aba4-2ac0fff8f217" path="/var/lib/kubelet/pods/11ce60c2-ca2f-49c7-aba4-2ac0fff8f217/volumes" Sep 11 15:13:01.632960 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:01.632927 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-177c81-h6v59"] Sep 11 15:13:01.633428 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:01.633306 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b95590c0-2f23-49ba-9d96-d18451d8b73c" containerName="revoke-keys" Sep 11 15:13:01.633428 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:01.633317 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b95590c0-2f23-49ba-9d96-d18451d8b73c" containerName="revoke-keys" Sep 11 15:13:01.633428 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:01.633324 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11ce60c2-ca2f-49c7-aba4-2ac0fff8f217" containerName="payload-pre-processing" Sep 11 15:13:01.633428 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:01.633330 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="11ce60c2-ca2f-49c7-aba4-2ac0fff8f217" containerName="payload-pre-processing" Sep 11 15:13:01.633428 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:01.633386 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b95590c0-2f23-49ba-9d96-d18451d8b73c" containerName="revoke-keys" Sep 11 15:13:01.633428 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:01.633395 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="11ce60c2-ca2f-49c7-aba4-2ac0fff8f217" containerName="payload-pre-processing" Sep 11 15:13:01.637843 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:01.637826 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-177c81-h6v59" Sep 11 15:13:01.640055 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:01.640034 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nks8w\"" Sep 11 15:13:01.640324 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:01.640308 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:13:01.644846 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:01.644823 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-177c81-h6v59"] Sep 11 15:13:01.721132 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:01.721064 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f23034-e808-4929-9980-c92ab45cd8eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-177c81-h6v59\" (UID: \"94f23034-e808-4929-9980-c92ab45cd8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-177c81-h6v59" Sep 11 15:13:01.821912 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:01.821884 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f23034-e808-4929-9980-c92ab45cd8eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-177c81-h6v59\" (UID: \"94f23034-e808-4929-9980-c92ab45cd8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-177c81-h6v59" Sep 11 15:13:01.822471 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:01.822453 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f23034-e808-4929-9980-c92ab45cd8eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-177c81-h6v59\" (UID: \"94f23034-e808-4929-9980-c92ab45cd8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-177c81-h6v59" Sep 11 15:13:01.949216 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:01.949130 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-177c81-h6v59" Sep 11 15:13:02.222890 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:02.222860 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-177c81-h6v59"] Sep 11 15:13:02.224387 ip-10-0-135-39 kubenswrapper[2583]: W0911 15:13:02.224350 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f23034_e808_4929_9980_c92ab45cd8eb.slice/crio-15f1ba5c646dd2b1204f12ba48d526ac61d6d51451c6aa752ae89ed777a3d230 WatchSource:0}: Error finding container 15f1ba5c646dd2b1204f12ba48d526ac61d6d51451c6aa752ae89ed777a3d230: Status 404 returned error can't find the container with id 15f1ba5c646dd2b1204f12ba48d526ac61d6d51451c6aa752ae89ed777a3d230 Sep 11 15:13:02.733920 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:02.733887 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-177c81-h6v59" event={"ID":"94f23034-e808-4929-9980-c92ab45cd8eb","Type":"ContainerStarted","Data":"af1a260135a62c898bde2ffe7054e002a814fb18cdaf1cc129a21612e30d6753"} Sep 11 15:13:02.734299 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:02.733943 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-177c81-h6v59" event={"ID":"94f23034-e808-4929-9980-c92ab45cd8eb","Type":"ContainerStarted","Data":"15f1ba5c646dd2b1204f12ba48d526ac61d6d51451c6aa752ae89ed777a3d230"} Sep 11 15:13:02.751501 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:02.751453 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-177c81-h6v59" podStartSLOduration=1.751434901 podStartE2EDuration="1.751434901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:13:02.749413174 +0000 UTC m=+1973.993398834" watchObservedRunningTime="2026-09-11 15:13:02.751434901 +0000 UTC m=+1973.995420608" Sep 11 15:13:03.741190 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:03.741158 2583 generic.go:356] "Generic (PLEG): container finished" podID="94f23034-e808-4929-9980-c92ab45cd8eb" containerID="af1a260135a62c898bde2ffe7054e002a814fb18cdaf1cc129a21612e30d6753" exitCode=0 Sep 11 15:13:03.741190 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:03.741195 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-177c81-h6v59" event={"ID":"94f23034-e808-4929-9980-c92ab45cd8eb","Type":"ContainerDied","Data":"af1a260135a62c898bde2ffe7054e002a814fb18cdaf1cc129a21612e30d6753"} Sep 11 15:13:04.881765 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:04.881742 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-177c81-h6v59" Sep 11 15:13:04.948293 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:04.948258 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f23034-e808-4929-9980-c92ab45cd8eb-maas-api-service-ca\") pod \"94f23034-e808-4929-9980-c92ab45cd8eb\" (UID: \"94f23034-e808-4929-9980-c92ab45cd8eb\") " Sep 11 15:13:04.948635 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:04.948614 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f23034-e808-4929-9980-c92ab45cd8eb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "94f23034-e808-4929-9980-c92ab45cd8eb" (UID: "94f23034-e808-4929-9980-c92ab45cd8eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:13:05.049227 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:05.049140 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f23034-e808-4929-9980-c92ab45cd8eb-maas-api-service-ca\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:13:05.754405 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:05.754362 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-177c81-h6v59" event={"ID":"94f23034-e808-4929-9980-c92ab45cd8eb","Type":"ContainerDied","Data":"15f1ba5c646dd2b1204f12ba48d526ac61d6d51451c6aa752ae89ed777a3d230"} Sep 11 15:13:05.754405 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:05.754401 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-177c81-h6v59" Sep 11 15:13:05.754689 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:05.754457 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15f1ba5c646dd2b1204f12ba48d526ac61d6d51451c6aa752ae89ed777a3d230" Sep 11 15:13:09.757218 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:09.757185 2583 scope.go:117] "RemoveContainer" containerID="e51fb3fc44ad6bb3e1a29a0776176cf409ef3244b814dfb41cc47c0744adfe09" Sep 11 15:13:09.766196 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:09.766174 2583 scope.go:117] "RemoveContainer" containerID="327825ae8e320cd5fb6beb3a50c9afdc703b3530361968893e40018296b3f4e8" Sep 11 15:13:09.774624 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:09.774603 2583 scope.go:117] "RemoveContainer" containerID="b993d9b7a64de20f8edce937df1b8e65ae5984b8f639a3a40cf3685fbaf40f82" Sep 11 15:13:17.393644 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:17.393603 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww"] Sep 11 15:13:17.462329 ip-10-0-135-39 kubenswrapper[2583]: E0911 15:13:17.462299 2583 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-ipp-a-177c81: configmap "payload-processing-plugins-e2e-ipp-a-177c81" not found Sep 11 15:13:17.462500 ip-10-0-135-39 kubenswrapper[2583]: E0911 15:13:17.462402 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c93ac22-6e88-4d5c-9afc-60cc97600e15-plugins-config-volume podName:7c93ac22-6e88-4d5c-9afc-60cc97600e15 nodeName:}" failed. No retries permitted until 2026-09-11 15:13:17.962386015 +0000 UTC m=+1989.206371653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/7c93ac22-6e88-4d5c-9afc-60cc97600e15-plugins-config-volume") pod "payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww" (UID: "7c93ac22-6e88-4d5c-9afc-60cc97600e15") : configmap "payload-processing-plugins-e2e-ipp-a-177c81" not found Sep 11 15:13:17.570512 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:17.570472 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-177c81-h6v59"] Sep 11 15:13:17.572795 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:17.572774 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-177c81-h6v59"] Sep 11 15:13:17.824915 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:17.824791 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww" podUID="7c93ac22-6e88-4d5c-9afc-60cc97600e15" containerName="payload-processing" containerID="cri-o://aa4aba040fb67f35e38747d31aa273c451329ee030bf523a1fd19fa5bf18117e" gracePeriod=30 Sep 11 15:13:17.967443 ip-10-0-135-39 kubenswrapper[2583]: E0911 15:13:17.967408 2583 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-ipp-a-177c81: configmap "payload-processing-plugins-e2e-ipp-a-177c81" not found Sep 11 15:13:17.967626 ip-10-0-135-39 kubenswrapper[2583]: E0911 15:13:17.967504 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c93ac22-6e88-4d5c-9afc-60cc97600e15-plugins-config-volume podName:7c93ac22-6e88-4d5c-9afc-60cc97600e15 nodeName:}" failed. No retries permitted until 2026-09-11 15:13:18.967486315 +0000 UTC m=+1990.211471953 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/7c93ac22-6e88-4d5c-9afc-60cc97600e15-plugins-config-volume") pod "payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww" (UID: "7c93ac22-6e88-4d5c-9afc-60cc97600e15") : configmap "payload-processing-plugins-e2e-ipp-a-177c81" not found Sep 11 15:13:18.083374 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:18.083307 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww" Sep 11 15:13:18.170066 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:18.170027 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dqsc\" (UniqueName: \"kubernetes.io/projected/7c93ac22-6e88-4d5c-9afc-60cc97600e15-kube-api-access-9dqsc\") pod \"7c93ac22-6e88-4d5c-9afc-60cc97600e15\" (UID: \"7c93ac22-6e88-4d5c-9afc-60cc97600e15\") " Sep 11 15:13:18.170240 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:18.170165 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c93ac22-6e88-4d5c-9afc-60cc97600e15-plugins-config-volume\") pod \"7c93ac22-6e88-4d5c-9afc-60cc97600e15\" (UID: \"7c93ac22-6e88-4d5c-9afc-60cc97600e15\") " Sep 11 15:13:18.170535 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:18.170509 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c93ac22-6e88-4d5c-9afc-60cc97600e15-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7c93ac22-6e88-4d5c-9afc-60cc97600e15" (UID: "7c93ac22-6e88-4d5c-9afc-60cc97600e15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:13:18.172274 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:18.172248 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c93ac22-6e88-4d5c-9afc-60cc97600e15-kube-api-access-9dqsc" (OuterVolumeSpecName: "kube-api-access-9dqsc") pod "7c93ac22-6e88-4d5c-9afc-60cc97600e15" (UID: "7c93ac22-6e88-4d5c-9afc-60cc97600e15"). InnerVolumeSpecName "kube-api-access-9dqsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:13:18.271229 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:18.271196 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dqsc\" (UniqueName: \"kubernetes.io/projected/7c93ac22-6e88-4d5c-9afc-60cc97600e15-kube-api-access-9dqsc\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:13:18.271229 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:18.271222 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c93ac22-6e88-4d5c-9afc-60cc97600e15-plugins-config-volume\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:13:18.831809 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:18.831768 2583 generic.go:356] "Generic (PLEG): container finished" podID="7c93ac22-6e88-4d5c-9afc-60cc97600e15" containerID="aa4aba040fb67f35e38747d31aa273c451329ee030bf523a1fd19fa5bf18117e" exitCode=0 Sep 11 15:13:18.831809 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:18.831813 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww" Sep 11 15:13:18.832267 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:18.831847 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww" event={"ID":"7c93ac22-6e88-4d5c-9afc-60cc97600e15","Type":"ContainerDied","Data":"aa4aba040fb67f35e38747d31aa273c451329ee030bf523a1fd19fa5bf18117e"} Sep 11 15:13:18.832267 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:18.831902 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww" event={"ID":"7c93ac22-6e88-4d5c-9afc-60cc97600e15","Type":"ContainerDied","Data":"1977170e136dec4a4d0bd1883f327a5b47a15349f089433aea4a0a851b94b6ad"} Sep 11 15:13:18.832267 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:18.831923 2583 scope.go:117] "RemoveContainer" containerID="aa4aba040fb67f35e38747d31aa273c451329ee030bf523a1fd19fa5bf18117e" Sep 11 15:13:18.842292 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:18.842273 2583 scope.go:117] "RemoveContainer" containerID="aa4aba040fb67f35e38747d31aa273c451329ee030bf523a1fd19fa5bf18117e" Sep 11 15:13:18.842600 ip-10-0-135-39 kubenswrapper[2583]: E0911 15:13:18.842579 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa4aba040fb67f35e38747d31aa273c451329ee030bf523a1fd19fa5bf18117e\": container with ID starting with aa4aba040fb67f35e38747d31aa273c451329ee030bf523a1fd19fa5bf18117e not found: ID does not exist" containerID="aa4aba040fb67f35e38747d31aa273c451329ee030bf523a1fd19fa5bf18117e" Sep 11 15:13:18.842687 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:18.842613 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa4aba040fb67f35e38747d31aa273c451329ee030bf523a1fd19fa5bf18117e"} err="failed to get container status \"aa4aba040fb67f35e38747d31aa273c451329ee030bf523a1fd19fa5bf18117e\": rpc error: code = NotFound desc = could not find container \"aa4aba040fb67f35e38747d31aa273c451329ee030bf523a1fd19fa5bf18117e\": container with ID starting with aa4aba040fb67f35e38747d31aa273c451329ee030bf523a1fd19fa5bf18117e not found: ID does not exist" Sep 11 15:13:18.854235 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:18.854211 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww"] Sep 11 15:13:18.857995 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:18.857968 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-177c81-576f87d755-l27ww"] Sep 11 15:13:19.330957 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:19.330923 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c93ac22-6e88-4d5c-9afc-60cc97600e15" path="/var/lib/kubelet/pods/7c93ac22-6e88-4d5c-9afc-60cc97600e15/volumes" Sep 11 15:13:19.331266 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:13:19.331253 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94f23034-e808-4929-9980-c92ab45cd8eb" path="/var/lib/kubelet/pods/94f23034-e808-4929-9980-c92ab45cd8eb/volumes" Sep 11 15:14:06.709545 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:06.709514 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee29a-h7k8f"] Sep 11 15:14:06.710012 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:06.709908 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c93ac22-6e88-4d5c-9afc-60cc97600e15" containerName="payload-processing" Sep 11 15:14:06.710012 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:06.709921 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c93ac22-6e88-4d5c-9afc-60cc97600e15" containerName="payload-processing" Sep 11 15:14:06.710012 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:06.709946 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94f23034-e808-4929-9980-c92ab45cd8eb" containerName="revoke-keys" Sep 11 15:14:06.710012 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:06.709951 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="94f23034-e808-4929-9980-c92ab45cd8eb" containerName="revoke-keys" Sep 11 15:14:06.710012 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:06.710004 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c93ac22-6e88-4d5c-9afc-60cc97600e15" containerName="payload-processing" Sep 11 15:14:06.710012 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:06.710010 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="94f23034-e808-4929-9980-c92ab45cd8eb" containerName="revoke-keys" Sep 11 15:14:06.713078 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:06.713058 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee29a-h7k8f" Sep 11 15:14:06.715375 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:06.715345 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:14:06.715502 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:06.715387 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nks8w\"" Sep 11 15:14:06.721673 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:06.721649 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee29a-h7k8f"] Sep 11 15:14:06.787120 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:06.787089 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86365521-2619-4712-babc-cfa79eb409d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4ee29a-h7k8f\" (UID: \"86365521-2619-4712-babc-cfa79eb409d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee29a-h7k8f" Sep 11 15:14:06.888590 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:06.888555 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86365521-2619-4712-babc-cfa79eb409d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4ee29a-h7k8f\" (UID: \"86365521-2619-4712-babc-cfa79eb409d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee29a-h7k8f" Sep 11 15:14:06.889330 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:06.889305 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86365521-2619-4712-babc-cfa79eb409d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4ee29a-h7k8f\" (UID: \"86365521-2619-4712-babc-cfa79eb409d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee29a-h7k8f" Sep 11 15:14:07.024839 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:07.024749 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee29a-h7k8f" Sep 11 15:14:07.302681 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:07.302653 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee29a-h7k8f"] Sep 11 15:14:07.304208 ip-10-0-135-39 kubenswrapper[2583]: W0911 15:14:07.304170 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86365521_2619_4712_babc_cfa79eb409d2.slice/crio-27fcad14752461a69f77e9997641f260e0d76f954957819520dd5acdbb0dd198 WatchSource:0}: Error finding container 27fcad14752461a69f77e9997641f260e0d76f954957819520dd5acdbb0dd198: Status 404 returned error can't find the container with id 27fcad14752461a69f77e9997641f260e0d76f954957819520dd5acdbb0dd198 Sep 11 15:14:08.116285 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:08.116250 2583 generic.go:356] "Generic (PLEG): container finished" podID="86365521-2619-4712-babc-cfa79eb409d2" containerID="503f9877f0493fb8e9a886d21ab08c5f5beb486faa81f3b39bd41d58a6b91791" exitCode=0 Sep 11 15:14:08.116739 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:08.116333 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee29a-h7k8f" event={"ID":"86365521-2619-4712-babc-cfa79eb409d2","Type":"ContainerDied","Data":"503f9877f0493fb8e9a886d21ab08c5f5beb486faa81f3b39bd41d58a6b91791"} Sep 11 15:14:08.116739 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:08.116380 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee29a-h7k8f" event={"ID":"86365521-2619-4712-babc-cfa79eb409d2","Type":"ContainerStarted","Data":"27fcad14752461a69f77e9997641f260e0d76f954957819520dd5acdbb0dd198"} Sep 11 15:14:09.251240 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:09.251213 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee29a-h7k8f" Sep 11 15:14:09.411791 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:09.411751 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86365521-2619-4712-babc-cfa79eb409d2-maas-api-service-ca\") pod \"86365521-2619-4712-babc-cfa79eb409d2\" (UID: \"86365521-2619-4712-babc-cfa79eb409d2\") " Sep 11 15:14:09.412185 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:09.412160 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86365521-2619-4712-babc-cfa79eb409d2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "86365521-2619-4712-babc-cfa79eb409d2" (UID: "86365521-2619-4712-babc-cfa79eb409d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:14:09.513283 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:09.513246 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86365521-2619-4712-babc-cfa79eb409d2-maas-api-service-ca\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:14:09.809888 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:09.809790 2583 scope.go:117] "RemoveContainer" containerID="6bdd5a0dd05e1c2fed3c7415009e87a2541c72df8331ee603d973f1ff022d697" Sep 11 15:14:10.129237 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:10.129205 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee29a-h7k8f" Sep 11 15:14:10.129403 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:10.129209 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee29a-h7k8f" event={"ID":"86365521-2619-4712-babc-cfa79eb409d2","Type":"ContainerDied","Data":"27fcad14752461a69f77e9997641f260e0d76f954957819520dd5acdbb0dd198"} Sep 11 15:14:10.129403 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:10.129324 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27fcad14752461a69f77e9997641f260e0d76f954957819520dd5acdbb0dd198" Sep 11 15:14:17.509257 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:17.509219 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf"] Sep 11 15:14:17.509842 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:17.509482 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf" podUID="b6507a31-c6b7-46f1-b022-6f54b72b1807" containerName="payload-processing" containerID="cri-o://6ca6433e5c7a9c2fb0a9d7eb28295326653c0ee9486fd104abc2dd7813611d13" gracePeriod=30 Sep 11 15:14:17.681063 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:17.681019 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee29a-h7k8f"] Sep 11 15:14:17.684789 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:17.684762 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ee29a-h7k8f"] Sep 11 15:14:17.767296 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:17.767240 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf" Sep 11 15:14:17.894353 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:17.894319 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvr4k\" (UniqueName: \"kubernetes.io/projected/b6507a31-c6b7-46f1-b022-6f54b72b1807-kube-api-access-dvr4k\") pod \"b6507a31-c6b7-46f1-b022-6f54b72b1807\" (UID: \"b6507a31-c6b7-46f1-b022-6f54b72b1807\") " Sep 11 15:14:17.894571 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:17.894411 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6507a31-c6b7-46f1-b022-6f54b72b1807-plugins-config-volume\") pod \"b6507a31-c6b7-46f1-b022-6f54b72b1807\" (UID: \"b6507a31-c6b7-46f1-b022-6f54b72b1807\") " Sep 11 15:14:17.894851 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:17.894827 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6507a31-c6b7-46f1-b022-6f54b72b1807-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b6507a31-c6b7-46f1-b022-6f54b72b1807" (UID: "b6507a31-c6b7-46f1-b022-6f54b72b1807"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:14:17.896463 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:17.896437 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6507a31-c6b7-46f1-b022-6f54b72b1807-kube-api-access-dvr4k" (OuterVolumeSpecName: "kube-api-access-dvr4k") pod "b6507a31-c6b7-46f1-b022-6f54b72b1807" (UID: "b6507a31-c6b7-46f1-b022-6f54b72b1807"). InnerVolumeSpecName "kube-api-access-dvr4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:14:17.996006 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:17.995961 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6507a31-c6b7-46f1-b022-6f54b72b1807-plugins-config-volume\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:14:17.996006 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:17.995999 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvr4k\" (UniqueName: \"kubernetes.io/projected/b6507a31-c6b7-46f1-b022-6f54b72b1807-kube-api-access-dvr4k\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:14:18.178057 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:18.178020 2583 generic.go:356] "Generic (PLEG): container finished" podID="b6507a31-c6b7-46f1-b022-6f54b72b1807" containerID="6ca6433e5c7a9c2fb0a9d7eb28295326653c0ee9486fd104abc2dd7813611d13" exitCode=0 Sep 11 15:14:18.178241 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:18.178066 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf" Sep 11 15:14:18.178241 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:18.178098 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf" event={"ID":"b6507a31-c6b7-46f1-b022-6f54b72b1807","Type":"ContainerDied","Data":"6ca6433e5c7a9c2fb0a9d7eb28295326653c0ee9486fd104abc2dd7813611d13"} Sep 11 15:14:18.178241 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:18.178153 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf" event={"ID":"b6507a31-c6b7-46f1-b022-6f54b72b1807","Type":"ContainerDied","Data":"608a5e168ae0781b06ac6c5a136168009e6257728ba55a8e14acfcd30d04778c"} Sep 11 15:14:18.178241 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:18.178178 2583 scope.go:117] "RemoveContainer" containerID="6ca6433e5c7a9c2fb0a9d7eb28295326653c0ee9486fd104abc2dd7813611d13" Sep 11 15:14:18.191689 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:18.191668 2583 scope.go:117] "RemoveContainer" containerID="6ca6433e5c7a9c2fb0a9d7eb28295326653c0ee9486fd104abc2dd7813611d13" Sep 11 15:14:18.192001 ip-10-0-135-39 kubenswrapper[2583]: E0911 15:14:18.191980 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ca6433e5c7a9c2fb0a9d7eb28295326653c0ee9486fd104abc2dd7813611d13\": container with ID starting with 6ca6433e5c7a9c2fb0a9d7eb28295326653c0ee9486fd104abc2dd7813611d13 not found: ID does not exist" containerID="6ca6433e5c7a9c2fb0a9d7eb28295326653c0ee9486fd104abc2dd7813611d13" Sep 11 15:14:18.192094 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:18.192011 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ca6433e5c7a9c2fb0a9d7eb28295326653c0ee9486fd104abc2dd7813611d13"} err="failed to get container status \"6ca6433e5c7a9c2fb0a9d7eb28295326653c0ee9486fd104abc2dd7813611d13\": rpc error: code = NotFound desc = could not find container \"6ca6433e5c7a9c2fb0a9d7eb28295326653c0ee9486fd104abc2dd7813611d13\": container with ID starting with 6ca6433e5c7a9c2fb0a9d7eb28295326653c0ee9486fd104abc2dd7813611d13 not found: ID does not exist" Sep 11 15:14:18.202914 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:18.202887 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf"] Sep 11 15:14:18.207314 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:18.207289 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4ee29a-68f45bf967-z6lhf"] Sep 11 15:14:19.330349 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:19.330313 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86365521-2619-4712-babc-cfa79eb409d2" path="/var/lib/kubelet/pods/86365521-2619-4712-babc-cfa79eb409d2/volumes" Sep 11 15:14:19.330763 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:19.330685 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6507a31-c6b7-46f1-b022-6f54b72b1807" path="/var/lib/kubelet/pods/b6507a31-c6b7-46f1-b022-6f54b72b1807/volumes" Sep 11 15:14:24.549374 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:24.549331 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-48898f-wp6mg"] Sep 11 15:14:24.549838 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:24.549749 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6507a31-c6b7-46f1-b022-6f54b72b1807" containerName="payload-processing" Sep 11 15:14:24.549838 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:24.549766 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6507a31-c6b7-46f1-b022-6f54b72b1807" containerName="payload-processing" Sep 11 15:14:24.549838 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:24.549776 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86365521-2619-4712-babc-cfa79eb409d2" containerName="revoke-keys" Sep 11 15:14:24.549838 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:24.549782 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="86365521-2619-4712-babc-cfa79eb409d2" containerName="revoke-keys" Sep 11 15:14:24.550002 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:24.549844 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="86365521-2619-4712-babc-cfa79eb409d2" containerName="revoke-keys" Sep 11 15:14:24.550002 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:24.549853 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6507a31-c6b7-46f1-b022-6f54b72b1807" containerName="payload-processing" Sep 11 15:14:24.554259 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:24.554235 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-48898f-wp6mg" Sep 11 15:14:24.556641 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:24.556616 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:14:24.556641 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:24.556626 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nks8w\"" Sep 11 15:14:24.558219 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:24.558189 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6876c1ac-795e-4f77-9a14-4112f2e5d1f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-48898f-wp6mg\" (UID: \"6876c1ac-795e-4f77-9a14-4112f2e5d1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-48898f-wp6mg" Sep 11 15:14:24.561671 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:24.561645 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-48898f-wp6mg"] Sep 11 15:14:24.659483 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:24.659434 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6876c1ac-795e-4f77-9a14-4112f2e5d1f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-48898f-wp6mg\" (UID: \"6876c1ac-795e-4f77-9a14-4112f2e5d1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-48898f-wp6mg" Sep 11 15:14:24.660089 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:24.660058 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6876c1ac-795e-4f77-9a14-4112f2e5d1f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-48898f-wp6mg\" (UID: \"6876c1ac-795e-4f77-9a14-4112f2e5d1f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-48898f-wp6mg" Sep 11 15:14:24.866213 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:24.866121 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-48898f-wp6mg" Sep 11 15:14:25.139376 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:25.139345 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-48898f-wp6mg"] Sep 11 15:14:25.141183 ip-10-0-135-39 kubenswrapper[2583]: W0911 15:14:25.141156 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6876c1ac_795e_4f77_9a14_4112f2e5d1f7.slice/crio-d363964c6245af02d2b1f0aaeb2deff59a82da2ec3a78ba13a1628a057115bfb WatchSource:0}: Error finding container d363964c6245af02d2b1f0aaeb2deff59a82da2ec3a78ba13a1628a057115bfb: Status 404 returned error can't find the container with id d363964c6245af02d2b1f0aaeb2deff59a82da2ec3a78ba13a1628a057115bfb Sep 11 15:14:25.221807 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:25.221771 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-48898f-wp6mg" event={"ID":"6876c1ac-795e-4f77-9a14-4112f2e5d1f7","Type":"ContainerStarted","Data":"d363964c6245af02d2b1f0aaeb2deff59a82da2ec3a78ba13a1628a057115bfb"} Sep 11 15:14:26.229412 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:26.229374 2583 generic.go:356] "Generic (PLEG): container finished" podID="6876c1ac-795e-4f77-9a14-4112f2e5d1f7" containerID="4ddf30927187cf5c5a1a7f97fd6ec604da9663f41d8c53b9f2eb6178baa8bb1f" exitCode=0 Sep 11 15:14:26.229855 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:26.229431 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-48898f-wp6mg" event={"ID":"6876c1ac-795e-4f77-9a14-4112f2e5d1f7","Type":"ContainerDied","Data":"4ddf30927187cf5c5a1a7f97fd6ec604da9663f41d8c53b9f2eb6178baa8bb1f"} Sep 11 15:14:27.371795 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:27.371763 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-48898f-wp6mg" Sep 11 15:14:27.481441 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:27.481396 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6876c1ac-795e-4f77-9a14-4112f2e5d1f7-maas-api-service-ca\") pod \"6876c1ac-795e-4f77-9a14-4112f2e5d1f7\" (UID: \"6876c1ac-795e-4f77-9a14-4112f2e5d1f7\") " Sep 11 15:14:27.481816 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:27.481791 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6876c1ac-795e-4f77-9a14-4112f2e5d1f7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6876c1ac-795e-4f77-9a14-4112f2e5d1f7" (UID: "6876c1ac-795e-4f77-9a14-4112f2e5d1f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:14:27.583000 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:27.582902 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6876c1ac-795e-4f77-9a14-4112f2e5d1f7-maas-api-service-ca\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:14:28.244265 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:28.244233 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-48898f-wp6mg" event={"ID":"6876c1ac-795e-4f77-9a14-4112f2e5d1f7","Type":"ContainerDied","Data":"d363964c6245af02d2b1f0aaeb2deff59a82da2ec3a78ba13a1628a057115bfb"} Sep 11 15:14:28.244455 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:28.244292 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d363964c6245af02d2b1f0aaeb2deff59a82da2ec3a78ba13a1628a057115bfb" Sep 11 15:14:28.244455 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:28.244247 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-48898f-wp6mg" Sep 11 15:14:35.056064 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:35.055979 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j"] Sep 11 15:14:35.056513 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:35.056317 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j" podUID="74156825-ef12-40fc-9862-e783ca3dd581" containerName="payload-pre-processing" containerID="cri-o://ef6aeab5cba947d6b872ec88e396561b4a637136e92da793ad955b82803a53b0" gracePeriod=30 Sep 11 15:14:35.220435 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:35.220380 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-48898f-wp6mg"] Sep 11 15:14:35.222761 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:35.222730 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-48898f-wp6mg"] Sep 11 15:14:35.285613 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:35.285574 2583 generic.go:356] "Generic (PLEG): container finished" podID="74156825-ef12-40fc-9862-e783ca3dd581" containerID="ef6aeab5cba947d6b872ec88e396561b4a637136e92da793ad955b82803a53b0" exitCode=0 Sep 11 15:14:35.285811 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:35.285647 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j" event={"ID":"74156825-ef12-40fc-9862-e783ca3dd581","Type":"ContainerDied","Data":"ef6aeab5cba947d6b872ec88e396561b4a637136e92da793ad955b82803a53b0"} Sep 11 15:14:35.314348 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:35.314277 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j" Sep 11 15:14:35.331213 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:35.331169 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6876c1ac-795e-4f77-9a14-4112f2e5d1f7" path="/var/lib/kubelet/pods/6876c1ac-795e-4f77-9a14-4112f2e5d1f7/volumes" Sep 11 15:14:35.360172 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:35.360134 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74156825-ef12-40fc-9862-e783ca3dd581-plugins-config-volume\") pod \"74156825-ef12-40fc-9862-e783ca3dd581\" (UID: \"74156825-ef12-40fc-9862-e783ca3dd581\") " Sep 11 15:14:35.360378 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:35.360196 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhrgh\" (UniqueName: \"kubernetes.io/projected/74156825-ef12-40fc-9862-e783ca3dd581-kube-api-access-mhrgh\") pod \"74156825-ef12-40fc-9862-e783ca3dd581\" (UID: \"74156825-ef12-40fc-9862-e783ca3dd581\") " Sep 11 15:14:35.360568 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:35.360539 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74156825-ef12-40fc-9862-e783ca3dd581-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "74156825-ef12-40fc-9862-e783ca3dd581" (UID: "74156825-ef12-40fc-9862-e783ca3dd581"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:14:35.362425 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:35.362396 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74156825-ef12-40fc-9862-e783ca3dd581-kube-api-access-mhrgh" (OuterVolumeSpecName: "kube-api-access-mhrgh") pod "74156825-ef12-40fc-9862-e783ca3dd581" (UID: "74156825-ef12-40fc-9862-e783ca3dd581"). InnerVolumeSpecName "kube-api-access-mhrgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:14:35.461052 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:35.461004 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhrgh\" (UniqueName: \"kubernetes.io/projected/74156825-ef12-40fc-9862-e783ca3dd581-kube-api-access-mhrgh\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:14:35.461052 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:35.461042 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74156825-ef12-40fc-9862-e783ca3dd581-plugins-config-volume\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:14:36.292599 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:36.292569 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j" Sep 11 15:14:36.293035 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:36.292562 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j" event={"ID":"74156825-ef12-40fc-9862-e783ca3dd581","Type":"ContainerDied","Data":"98b2bc01d16b72be372c8268881d476714aeb5be395931c0c7a8ac8dd189a369"} Sep 11 15:14:36.293035 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:36.292734 2583 scope.go:117] "RemoveContainer" containerID="ef6aeab5cba947d6b872ec88e396561b4a637136e92da793ad955b82803a53b0" Sep 11 15:14:36.315912 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:36.315881 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j"] Sep 11 15:14:36.319605 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:36.319576 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-48898f-f4db79966-rsv9j"] Sep 11 15:14:37.330902 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:37.330869 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74156825-ef12-40fc-9862-e783ca3dd581" path="/var/lib/kubelet/pods/74156825-ef12-40fc-9862-e783ca3dd581/volumes" Sep 11 15:14:42.160699 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:42.160663 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5"] Sep 11 15:14:42.161064 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:42.160895 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" podUID="f03699c7-057d-4bce-a174-a5784cfcef79" containerName="istio-proxy" containerID="cri-o://0789048a0f6391be4211a9930756fa9badab9aaf8d33af4db9c18b455f490041" gracePeriod=30 Sep 11 15:14:46.386167 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:46.386125 2583 patch_prober.go:28] interesting pod/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 15:14:46.386538 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:46.386182 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" podUID="f03699c7-057d-4bce-a174-a5784cfcef79" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 15:14:47.358598 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.358569 2583 generic.go:356] "Generic (PLEG): container finished" podID="f03699c7-057d-4bce-a174-a5784cfcef79" containerID="0789048a0f6391be4211a9930756fa9badab9aaf8d33af4db9c18b455f490041" exitCode=0 Sep 11 15:14:47.358742 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.358656 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" event={"ID":"f03699c7-057d-4bce-a174-a5784cfcef79","Type":"ContainerDied","Data":"0789048a0f6391be4211a9930756fa9badab9aaf8d33af4db9c18b455f490041"} Sep 11 15:14:47.358809 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.358747 2583 scope.go:117] "RemoveContainer" containerID="856d49a793f270c8c562dcf4b8bc4a46e4143a9d5a18d6d0b246d399323eec0e" Sep 11 15:14:47.410157 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.410136 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:14:47.473792 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.473726 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-credential-socket\") pod \"f03699c7-057d-4bce-a174-a5784cfcef79\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " Sep 11 15:14:47.473792 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.473786 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7g78j\" (UniqueName: \"kubernetes.io/projected/f03699c7-057d-4bce-a174-a5784cfcef79-kube-api-access-7g78j\") pod \"f03699c7-057d-4bce-a174-a5784cfcef79\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " Sep 11 15:14:47.473969 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.473805 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-istio-data\") pod \"f03699c7-057d-4bce-a174-a5784cfcef79\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " Sep 11 15:14:47.473969 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.473833 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-workload-certs\") pod \"f03699c7-057d-4bce-a174-a5784cfcef79\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " Sep 11 15:14:47.473969 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.473873 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f03699c7-057d-4bce-a174-a5784cfcef79-istio-podinfo\") pod \"f03699c7-057d-4bce-a174-a5784cfcef79\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " Sep 11 15:14:47.473969 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.473903 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-istio-envoy\") pod \"f03699c7-057d-4bce-a174-a5784cfcef79\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " Sep 11 15:14:47.473969 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.473926 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f03699c7-057d-4bce-a174-a5784cfcef79-istio-token\") pod \"f03699c7-057d-4bce-a174-a5784cfcef79\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " Sep 11 15:14:47.473969 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.473964 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-workload-socket\") pod \"f03699c7-057d-4bce-a174-a5784cfcef79\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " Sep 11 15:14:47.474254 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.473985 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f03699c7-057d-4bce-a174-a5784cfcef79" (UID: "f03699c7-057d-4bce-a174-a5784cfcef79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:14:47.474254 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.473994 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f03699c7-057d-4bce-a174-a5784cfcef79-istiod-ca-cert\") pod \"f03699c7-057d-4bce-a174-a5784cfcef79\" (UID: \"f03699c7-057d-4bce-a174-a5784cfcef79\") " Sep 11 15:14:47.474254 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.474099 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-istio-data" (OuterVolumeSpecName: "istio-data") pod "f03699c7-057d-4bce-a174-a5784cfcef79" (UID: "f03699c7-057d-4bce-a174-a5784cfcef79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:14:47.474254 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.474187 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f03699c7-057d-4bce-a174-a5784cfcef79" (UID: "f03699c7-057d-4bce-a174-a5784cfcef79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:14:47.474454 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.474378 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-istio-data\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:14:47.474454 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.474400 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-workload-certs\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:14:47.474454 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.474413 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-credential-socket\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:14:47.474454 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.474444 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f03699c7-057d-4bce-a174-a5784cfcef79-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f03699c7-057d-4bce-a174-a5784cfcef79" (UID: "f03699c7-057d-4bce-a174-a5784cfcef79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:14:47.474690 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.474667 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f03699c7-057d-4bce-a174-a5784cfcef79" (UID: "f03699c7-057d-4bce-a174-a5784cfcef79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:14:47.476166 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.476132 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f03699c7-057d-4bce-a174-a5784cfcef79-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f03699c7-057d-4bce-a174-a5784cfcef79" (UID: "f03699c7-057d-4bce-a174-a5784cfcef79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 15:14:47.476258 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.476232 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f03699c7-057d-4bce-a174-a5784cfcef79" (UID: "f03699c7-057d-4bce-a174-a5784cfcef79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:14:47.476312 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.476246 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f03699c7-057d-4bce-a174-a5784cfcef79-istio-token" (OuterVolumeSpecName: "istio-token") pod "f03699c7-057d-4bce-a174-a5784cfcef79" (UID: "f03699c7-057d-4bce-a174-a5784cfcef79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:14:47.476312 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.476256 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f03699c7-057d-4bce-a174-a5784cfcef79-kube-api-access-7g78j" (OuterVolumeSpecName: "kube-api-access-7g78j") pod "f03699c7-057d-4bce-a174-a5784cfcef79" (UID: "f03699c7-057d-4bce-a174-a5784cfcef79"). InnerVolumeSpecName "kube-api-access-7g78j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:14:47.575802 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.575775 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g78j\" (UniqueName: \"kubernetes.io/projected/f03699c7-057d-4bce-a174-a5784cfcef79-kube-api-access-7g78j\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:14:47.575802 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.575799 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f03699c7-057d-4bce-a174-a5784cfcef79-istio-podinfo\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:14:47.575966 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.575808 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-istio-envoy\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:14:47.575966 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.575816 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f03699c7-057d-4bce-a174-a5784cfcef79-istio-token\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:14:47.575966 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.575823 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03699c7-057d-4bce-a174-a5784cfcef79-workload-socket\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:14:47.575966 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:47.575831 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f03699c7-057d-4bce-a174-a5784cfcef79-istiod-ca-cert\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:14:48.365984 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:48.365951 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" event={"ID":"f03699c7-057d-4bce-a174-a5784cfcef79","Type":"ContainerDied","Data":"debb1ec839f707eabc8d37b50823d5ca4f2b563103f614e6a581a0451796456b"} Sep 11 15:14:48.366169 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:48.365999 2583 scope.go:117] "RemoveContainer" containerID="0789048a0f6391be4211a9930756fa9badab9aaf8d33af4db9c18b455f490041" Sep 11 15:14:48.366169 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:48.365966 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5" Sep 11 15:14:48.387588 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:48.387561 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5"] Sep 11 15:14:48.392054 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:48.392034 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-48898f-openshift-default-c6c9586f8-4kxk5"] Sep 11 15:14:49.335957 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:49.332054 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f03699c7-057d-4bce-a174-a5784cfcef79" path="/var/lib/kubelet/pods/f03699c7-057d-4bce-a174-a5784cfcef79/volumes" Sep 11 15:14:53.027551 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.027504 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6"] Sep 11 15:14:53.028333 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.028265 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6" podUID="d613d4c4-e1cc-428c-badd-4fb01bee8c54" containerName="payload-processing" containerID="cri-o://4a769ae15d45c2212e57a5dd59aa9688fd95170253a65d75f01e96ab031bd90a" gracePeriod=30 Sep 11 15:14:53.030813 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.030789 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5"] Sep 11 15:14:53.031048 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.030997 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5" podUID="08a86b3b-ce50-4d76-8344-af31d35d178f" containerName="payload-pre-processing" containerID="cri-o://6888b5b36eeda00d5a15f4b80307adb3f158c8bf5dcaeff12088e236f378cce0" gracePeriod=30 Sep 11 15:14:53.286141 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.286114 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6" Sep 11 15:14:53.295609 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.295591 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5" Sep 11 15:14:53.394509 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.394477 2583 generic.go:356] "Generic (PLEG): container finished" podID="d613d4c4-e1cc-428c-badd-4fb01bee8c54" containerID="4a769ae15d45c2212e57a5dd59aa9688fd95170253a65d75f01e96ab031bd90a" exitCode=0 Sep 11 15:14:53.394696 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.394520 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6" Sep 11 15:14:53.394696 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.394566 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6" event={"ID":"d613d4c4-e1cc-428c-badd-4fb01bee8c54","Type":"ContainerDied","Data":"4a769ae15d45c2212e57a5dd59aa9688fd95170253a65d75f01e96ab031bd90a"} Sep 11 15:14:53.394696 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.394618 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6" event={"ID":"d613d4c4-e1cc-428c-badd-4fb01bee8c54","Type":"ContainerDied","Data":"7848f10e07a0595b124ef5e11db3c653e9b16f2d1fe01f98a8393a111d1cd3fa"} Sep 11 15:14:53.394696 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.394638 2583 scope.go:117] "RemoveContainer" containerID="4a769ae15d45c2212e57a5dd59aa9688fd95170253a65d75f01e96ab031bd90a" Sep 11 15:14:53.395941 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.395921 2583 generic.go:356] "Generic (PLEG): container finished" podID="08a86b3b-ce50-4d76-8344-af31d35d178f" containerID="6888b5b36eeda00d5a15f4b80307adb3f158c8bf5dcaeff12088e236f378cce0" exitCode=0 Sep 11 15:14:53.396055 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.395948 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5" event={"ID":"08a86b3b-ce50-4d76-8344-af31d35d178f","Type":"ContainerDied","Data":"6888b5b36eeda00d5a15f4b80307adb3f158c8bf5dcaeff12088e236f378cce0"} Sep 11 15:14:53.396055 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.395961 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5" Sep 11 15:14:53.396055 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.395990 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5" event={"ID":"08a86b3b-ce50-4d76-8344-af31d35d178f","Type":"ContainerDied","Data":"f894833bad26605cde30f28436b2a9f151fb1dabbe15bfbcee3024b59d908226"} Sep 11 15:14:53.403337 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.403319 2583 scope.go:117] "RemoveContainer" containerID="4a769ae15d45c2212e57a5dd59aa9688fd95170253a65d75f01e96ab031bd90a" Sep 11 15:14:53.403611 ip-10-0-135-39 kubenswrapper[2583]: E0911 15:14:53.403594 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a769ae15d45c2212e57a5dd59aa9688fd95170253a65d75f01e96ab031bd90a\": container with ID starting with 4a769ae15d45c2212e57a5dd59aa9688fd95170253a65d75f01e96ab031bd90a not found: ID does not exist" containerID="4a769ae15d45c2212e57a5dd59aa9688fd95170253a65d75f01e96ab031bd90a" Sep 11 15:14:53.403659 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.403619 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a769ae15d45c2212e57a5dd59aa9688fd95170253a65d75f01e96ab031bd90a"} err="failed to get container status \"4a769ae15d45c2212e57a5dd59aa9688fd95170253a65d75f01e96ab031bd90a\": rpc error: code = NotFound desc = could not find container \"4a769ae15d45c2212e57a5dd59aa9688fd95170253a65d75f01e96ab031bd90a\": container with ID starting with 4a769ae15d45c2212e57a5dd59aa9688fd95170253a65d75f01e96ab031bd90a not found: ID does not exist" Sep 11 15:14:53.403659 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.403636 2583 scope.go:117] "RemoveContainer" containerID="6888b5b36eeda00d5a15f4b80307adb3f158c8bf5dcaeff12088e236f378cce0" Sep 11 15:14:53.411387 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.411366 2583 scope.go:117] "RemoveContainer" containerID="6888b5b36eeda00d5a15f4b80307adb3f158c8bf5dcaeff12088e236f378cce0" Sep 11 15:14:53.411626 ip-10-0-135-39 kubenswrapper[2583]: E0911 15:14:53.411612 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6888b5b36eeda00d5a15f4b80307adb3f158c8bf5dcaeff12088e236f378cce0\": container with ID starting with 6888b5b36eeda00d5a15f4b80307adb3f158c8bf5dcaeff12088e236f378cce0 not found: ID does not exist" containerID="6888b5b36eeda00d5a15f4b80307adb3f158c8bf5dcaeff12088e236f378cce0" Sep 11 15:14:53.411674 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.411631 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6888b5b36eeda00d5a15f4b80307adb3f158c8bf5dcaeff12088e236f378cce0"} err="failed to get container status \"6888b5b36eeda00d5a15f4b80307adb3f158c8bf5dcaeff12088e236f378cce0\": rpc error: code = NotFound desc = could not find container \"6888b5b36eeda00d5a15f4b80307adb3f158c8bf5dcaeff12088e236f378cce0\": container with ID starting with 6888b5b36eeda00d5a15f4b80307adb3f158c8bf5dcaeff12088e236f378cce0 not found: ID does not exist" Sep 11 15:14:53.423913 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.423895 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d613d4c4-e1cc-428c-badd-4fb01bee8c54-plugins-config-volume\") pod \"d613d4c4-e1cc-428c-badd-4fb01bee8c54\" (UID: \"d613d4c4-e1cc-428c-badd-4fb01bee8c54\") " Sep 11 15:14:53.423989 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.423963 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgklp\" (UniqueName: \"kubernetes.io/projected/08a86b3b-ce50-4d76-8344-af31d35d178f-kube-api-access-lgklp\") pod \"08a86b3b-ce50-4d76-8344-af31d35d178f\" (UID: \"08a86b3b-ce50-4d76-8344-af31d35d178f\") " Sep 11 15:14:53.424027 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.423988 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08a86b3b-ce50-4d76-8344-af31d35d178f-plugins-config-volume\") pod \"08a86b3b-ce50-4d76-8344-af31d35d178f\" (UID: \"08a86b3b-ce50-4d76-8344-af31d35d178f\") " Sep 11 15:14:53.424062 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.424032 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fznbq\" (UniqueName: \"kubernetes.io/projected/d613d4c4-e1cc-428c-badd-4fb01bee8c54-kube-api-access-fznbq\") pod \"d613d4c4-e1cc-428c-badd-4fb01bee8c54\" (UID: \"d613d4c4-e1cc-428c-badd-4fb01bee8c54\") " Sep 11 15:14:53.424328 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.424303 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08a86b3b-ce50-4d76-8344-af31d35d178f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "08a86b3b-ce50-4d76-8344-af31d35d178f" (UID: "08a86b3b-ce50-4d76-8344-af31d35d178f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:14:53.424400 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.424300 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d613d4c4-e1cc-428c-badd-4fb01bee8c54-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d613d4c4-e1cc-428c-badd-4fb01bee8c54" (UID: "d613d4c4-e1cc-428c-badd-4fb01bee8c54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:14:53.425933 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.425911 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a86b3b-ce50-4d76-8344-af31d35d178f-kube-api-access-lgklp" (OuterVolumeSpecName: "kube-api-access-lgklp") pod "08a86b3b-ce50-4d76-8344-af31d35d178f" (UID: "08a86b3b-ce50-4d76-8344-af31d35d178f"). InnerVolumeSpecName "kube-api-access-lgklp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:14:53.426024 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.426007 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d613d4c4-e1cc-428c-badd-4fb01bee8c54-kube-api-access-fznbq" (OuterVolumeSpecName: "kube-api-access-fznbq") pod "d613d4c4-e1cc-428c-badd-4fb01bee8c54" (UID: "d613d4c4-e1cc-428c-badd-4fb01bee8c54"). InnerVolumeSpecName "kube-api-access-fznbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:14:53.525547 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.525511 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d613d4c4-e1cc-428c-badd-4fb01bee8c54-plugins-config-volume\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:14:53.525547 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.525544 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgklp\" (UniqueName: \"kubernetes.io/projected/08a86b3b-ce50-4d76-8344-af31d35d178f-kube-api-access-lgklp\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:14:53.525547 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.525553 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08a86b3b-ce50-4d76-8344-af31d35d178f-plugins-config-volume\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:14:53.525749 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.525562 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fznbq\" (UniqueName: \"kubernetes.io/projected/d613d4c4-e1cc-428c-badd-4fb01bee8c54-kube-api-access-fznbq\") on node \"ip-10-0-135-39.ec2.internal\" DevicePath \"\"" Sep 11 15:14:53.716854 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.716829 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6"] Sep 11 15:14:53.721234 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.721207 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-928943-5f8c7db995-snll6"] Sep 11 15:14:53.730805 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.730782 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5"] Sep 11 15:14:53.734387 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:53.734366 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-928943-656956f754-zj6v5"] Sep 11 15:14:55.330720 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:55.330674 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08a86b3b-ce50-4d76-8344-af31d35d178f" path="/var/lib/kubelet/pods/08a86b3b-ce50-4d76-8344-af31d35d178f/volumes" Sep 11 15:14:55.331085 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:14:55.331051 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d613d4c4-e1cc-428c-badd-4fb01bee8c54" path="/var/lib/kubelet/pods/d613d4c4-e1cc-428c-badd-4fb01bee8c54/volumes" Sep 11 15:15:00.136490 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:00.136452 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818995-wcvc2"] Sep 11 15:15:00.136896 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:00.136842 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a86b3b-ce50-4d76-8344-af31d35d178f" containerName="payload-pre-processing" Sep 11 15:15:00.136896 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:00.136855 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a86b3b-ce50-4d76-8344-af31d35d178f" containerName="payload-pre-processing" Sep 11 15:15:00.136896 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:00.136867 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d613d4c4-e1cc-428c-badd-4fb01bee8c54" containerName="payload-processing" Sep 11 15:15:00.136896 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:00.136873 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d613d4c4-e1cc-428c-badd-4fb01bee8c54" containerName="payload-processing" Sep 11 15:15:00.136896 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:00.136880 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6876c1ac-795e-4f77-9a14-4112f2e5d1f7" containerName="revoke-keys" Sep 11 15:15:00.136896 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:00.136886 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6876c1ac-795e-4f77-9a14-4112f2e5d1f7" containerName="revoke-keys" Sep 11 15:15:00.136896 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:00.136898 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f03699c7-057d-4bce-a174-a5784cfcef79" containerName="istio-proxy" Sep 11 15:15:00.137123 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:00.136903 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f03699c7-057d-4bce-a174-a5784cfcef79" containerName="istio-proxy" Sep 11 15:15:00.137123 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:00.136912 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74156825-ef12-40fc-9862-e783ca3dd581" containerName="payload-pre-processing" Sep 11 15:15:00.137123 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:00.136918 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="74156825-ef12-40fc-9862-e783ca3dd581" containerName="payload-pre-processing" Sep 11 15:15:00.137123 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:00.136927 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f03699c7-057d-4bce-a174-a5784cfcef79" containerName="istio-proxy" Sep 11 15:15:00.137123 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:00.136932 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f03699c7-057d-4bce-a174-a5784cfcef79" containerName="istio-proxy" Sep 11 15:15:00.137123 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:00.136989 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="08a86b3b-ce50-4d76-8344-af31d35d178f" containerName="payload-pre-processing" Sep 11 15:15:00.137123 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:00.136998 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="74156825-ef12-40fc-9862-e783ca3dd581" containerName="payload-pre-processing" Sep 11 15:15:00.137123 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:00.137003 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d613d4c4-e1cc-428c-badd-4fb01bee8c54" containerName="payload-processing" Sep 11 15:15:00.137123 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:00.137009 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f03699c7-057d-4bce-a174-a5784cfcef79" containerName="istio-proxy" Sep 11 15:15:00.137123 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:00.137014 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6876c1ac-795e-4f77-9a14-4112f2e5d1f7" containerName="revoke-keys" Sep 11 15:15:00.141266 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:00.141246 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818995-wcvc2" Sep 11 15:15:00.143860 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:00.143831 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nks8w\"" Sep 11 15:15:00.146529 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:00.146501 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818995-wcvc2"] Sep 11 15:15:00.153532 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:00.153506 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818995-wcvc2" Sep 11 15:15:00.423569 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:00.423545 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818995-wcvc2"] Sep 11 15:15:00.425194 ip-10-0-135-39 kubenswrapper[2583]: W0911 15:15:00.425165 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f384225_3ff7_48ed_a514_8b671840b96d.slice/crio-ceef8f4b28b4402b477fb3f5e120143e03e6dcf8c5ce228c09697de032cd7e4f WatchSource:0}: Error finding container ceef8f4b28b4402b477fb3f5e120143e03e6dcf8c5ce228c09697de032cd7e4f: Status 404 returned error can't find the container with id ceef8f4b28b4402b477fb3f5e120143e03e6dcf8c5ce228c09697de032cd7e4f Sep 11 15:15:00.437844 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:00.437816 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818995-wcvc2" event={"ID":"3f384225-3ff7-48ed-a514-8b671840b96d","Type":"ContainerStarted","Data":"ceef8f4b28b4402b477fb3f5e120143e03e6dcf8c5ce228c09697de032cd7e4f"} Sep 11 15:15:01.444506 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:01.444469 2583 generic.go:356] "Generic (PLEG): container finished" podID="3f384225-3ff7-48ed-a514-8b671840b96d" containerID="7910b22140390ac16724b2f9d9292bac395ebc33bf99e8589c39fa83f5cbc52f" exitCode=0 Sep 11 15:15:01.444923 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:01.444513 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818995-wcvc2" event={"ID":"3f384225-3ff7-48ed-a514-8b671840b96d","Type":"ContainerDied","Data":"7910b22140390ac16724b2f9d9292bac395ebc33bf99e8589c39fa83f5cbc52f"} Sep 11 15:15:02.573886 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:02.573862 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818995-wcvc2" Sep 11 15:15:03.457111 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:03.457075 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818995-wcvc2" event={"ID":"3f384225-3ff7-48ed-a514-8b671840b96d","Type":"ContainerDied","Data":"ceef8f4b28b4402b477fb3f5e120143e03e6dcf8c5ce228c09697de032cd7e4f"} Sep 11 15:15:03.457262 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:03.457124 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ceef8f4b28b4402b477fb3f5e120143e03e6dcf8c5ce228c09697de032cd7e4f" Sep 11 15:15:03.457262 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:03.457090 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818995-wcvc2" Sep 11 15:15:09.380218 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:09.380181 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lwrt2_57d4ba26-384d-42f8-86a1-335e27cdf762/console-operator/1.log" Sep 11 15:15:09.386943 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:15:09.386914 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lwrt2_57d4ba26-384d-42f8-86a1-335e27cdf762/console-operator/1.log" Sep 11 15:19:09.894514 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:19:09.894480 2583 scope.go:117] "RemoveContainer" containerID="af1a260135a62c898bde2ffe7054e002a814fb18cdaf1cc129a21612e30d6753" Sep 11 15:19:09.903031 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:19:09.903013 2583 scope.go:117] "RemoveContainer" containerID="0612f85964d9dc7c453f69735ac4b2ab1687cd74c769f85e3ae400f9a3c75e18" Sep 11 15:20:09.414563 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:20:09.414532 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lwrt2_57d4ba26-384d-42f8-86a1-335e27cdf762/console-operator/1.log" Sep 11 15:20:09.423414 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:20:09.423387 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lwrt2_57d4ba26-384d-42f8-86a1-335e27cdf762/console-operator/1.log" Sep 11 15:20:09.928714 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:20:09.928684 2583 scope.go:117] "RemoveContainer" containerID="503f9877f0493fb8e9a886d21ab08c5f5beb486faa81f3b39bd41d58a6b91791" Sep 11 15:21:09.949359 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:09.949327 2583 scope.go:117] "RemoveContainer" containerID="4ddf30927187cf5c5a1a7f97fd6ec604da9663f41d8c53b9f2eb6178baa8bb1f" Sep 11 15:21:17.518394 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:17.518363 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-q9lp8_a9bb7c91-9123-4dc4-a469-7b7af81d9d4d/manager/0.log" Sep 11 15:21:17.923370 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:17.923340 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-s7722_ddc27eeb-1fbb-4110-a45d-c255ed11282c/server/0.log" Sep 11 15:21:18.056140 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:18.056097 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-r846f_ab41be90-a53b-4678-8c6b-9f3f8ea30dbf/manager/0.log" Sep 11 15:21:18.437548 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:18.437520 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67f987f856-vt2v2_63e3a376-f5d7-44f7-b725-e2f4ec532edc/copy-manifests/0.log" Sep 11 15:21:18.455661 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:18.455632 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67f987f856-vt2v2_63e3a376-f5d7-44f7-b725-e2f4ec532edc/manager/0.log" Sep 11 15:21:19.062848 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:19.062816 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29818980-7fctb_86e8255c-0236-46e7-8bb8-c3ab6fde2d8b/cleanup/0.log" Sep 11 15:21:19.173887 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:19.173858 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29818995-wcvc2_3f384225-3ff7-48ed-a514-8b671840b96d/cleanup/0.log" Sep 11 15:21:20.426467 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:20.426389 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-pxbzq_2264c0d5-92e6-432f-8e68-26de9c9c3b5a/manager/0.log" Sep 11 15:21:20.550886 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:20.550858 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-kl8dk_70c7f356-3f7a-4b55-b032-4165e1864109/kuadrant-console-plugin/0.log" Sep 11 15:21:20.957069 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:20.957035 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-gddvn_3d239b94-d5e2-47ee-9800-285917fcc7f4/manager/0.log" Sep 11 15:21:21.357193 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:21.357118 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h_fe579da6-e7a4-4a75-aea1-c2957e757fa9/istio-proxy/0.log" Sep 11 15:21:21.915307 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:21.915279 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-rp6sn_b6775488-0433-4467-a80a-db83b0ca7de9/istio-proxy/1.log" Sep 11 15:21:22.057491 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:22.057462 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-xc4hx_b5e55d5d-7bdf-4a23-af47-fefd40e063d4/payload-pre-processing/0.log" Sep 11 15:21:22.945131 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:22.945101 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-krjdb_848c1889-60dc-4d88-b467-a1c51f9bf3b2/main/0.log" Sep 11 15:21:23.190109 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:23.190082 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqclqt_e4acab8e-4026-4d69-9820-1165df545af3/main/0.log" Sep 11 15:21:23.319322 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:23.319239 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-jjrd8_23eb9fe9-3dac-4380-a381-d0f08e595303/main/0.log" Sep 11 15:21:29.868153 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:29.868120 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-slnpk/must-gather-86wv8"] Sep 11 15:21:29.868528 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:29.868458 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f384225-3ff7-48ed-a514-8b671840b96d" containerName="cleanup" Sep 11 15:21:29.868528 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:29.868472 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f384225-3ff7-48ed-a514-8b671840b96d" containerName="cleanup" Sep 11 15:21:29.868528 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:29.868525 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f384225-3ff7-48ed-a514-8b671840b96d" containerName="cleanup" Sep 11 15:21:29.868780 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:29.868537 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f03699c7-057d-4bce-a174-a5784cfcef79" containerName="istio-proxy" Sep 11 15:21:29.871838 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:29.871817 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-slnpk/must-gather-86wv8" Sep 11 15:21:29.874334 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:29.874313 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-slnpk\"/\"openshift-service-ca.crt\"" Sep 11 15:21:29.875078 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:29.875061 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-slnpk\"/\"default-dockercfg-w6qf6\"" Sep 11 15:21:29.875175 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:29.875075 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-slnpk\"/\"kube-root-ca.crt\"" Sep 11 15:21:29.881552 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:29.881529 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-slnpk/must-gather-86wv8"] Sep 11 15:21:30.028998 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:30.028968 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a24cd97c-0a48-402d-8d83-70e4d45ddc57-must-gather-output\") pod \"must-gather-86wv8\" (UID: \"a24cd97c-0a48-402d-8d83-70e4d45ddc57\") " pod="openshift-must-gather-slnpk/must-gather-86wv8" Sep 11 15:21:30.029153 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:30.029026 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9mrk\" (UniqueName: \"kubernetes.io/projected/a24cd97c-0a48-402d-8d83-70e4d45ddc57-kube-api-access-l9mrk\") pod \"must-gather-86wv8\" (UID: \"a24cd97c-0a48-402d-8d83-70e4d45ddc57\") " pod="openshift-must-gather-slnpk/must-gather-86wv8" Sep 11 15:21:30.129876 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:30.129845 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a24cd97c-0a48-402d-8d83-70e4d45ddc57-must-gather-output\") pod \"must-gather-86wv8\" (UID: \"a24cd97c-0a48-402d-8d83-70e4d45ddc57\") " pod="openshift-must-gather-slnpk/must-gather-86wv8" Sep 11 15:21:30.130016 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:30.129935 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9mrk\" (UniqueName: \"kubernetes.io/projected/a24cd97c-0a48-402d-8d83-70e4d45ddc57-kube-api-access-l9mrk\") pod \"must-gather-86wv8\" (UID: \"a24cd97c-0a48-402d-8d83-70e4d45ddc57\") " pod="openshift-must-gather-slnpk/must-gather-86wv8" Sep 11 15:21:30.130230 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:30.130212 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a24cd97c-0a48-402d-8d83-70e4d45ddc57-must-gather-output\") pod \"must-gather-86wv8\" (UID: \"a24cd97c-0a48-402d-8d83-70e4d45ddc57\") " pod="openshift-must-gather-slnpk/must-gather-86wv8" Sep 11 15:21:30.137609 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:30.137579 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9mrk\" (UniqueName: \"kubernetes.io/projected/a24cd97c-0a48-402d-8d83-70e4d45ddc57-kube-api-access-l9mrk\") pod \"must-gather-86wv8\" (UID: \"a24cd97c-0a48-402d-8d83-70e4d45ddc57\") " pod="openshift-must-gather-slnpk/must-gather-86wv8" Sep 11 15:21:30.182191 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:30.182164 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-slnpk/must-gather-86wv8" Sep 11 15:21:30.454005 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:30.453980 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-slnpk/must-gather-86wv8"] Sep 11 15:21:30.455752 ip-10-0-135-39 kubenswrapper[2583]: W0911 15:21:30.455724 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda24cd97c_0a48_402d_8d83_70e4d45ddc57.slice/crio-1c0d580da64fb1b9c60646c1025f1a32292156f1b8edce08d31a7737f0b88fef WatchSource:0}: Error finding container 1c0d580da64fb1b9c60646c1025f1a32292156f1b8edce08d31a7737f0b88fef: Status 404 returned error can't find the container with id 1c0d580da64fb1b9c60646c1025f1a32292156f1b8edce08d31a7737f0b88fef Sep 11 15:21:30.457551 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:30.457535 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 15:21:30.590154 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:30.590120 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-slnpk/must-gather-86wv8" event={"ID":"a24cd97c-0a48-402d-8d83-70e4d45ddc57","Type":"ContainerStarted","Data":"1c0d580da64fb1b9c60646c1025f1a32292156f1b8edce08d31a7737f0b88fef"} Sep 11 15:21:31.602047 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:31.602007 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-slnpk/must-gather-86wv8" event={"ID":"a24cd97c-0a48-402d-8d83-70e4d45ddc57","Type":"ContainerStarted","Data":"c5c1b4040e07862bfe6553701b6ceaf98971fc0cbdbe1b00284731d910d18a7d"} Sep 11 15:21:32.617768 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:32.616834 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-slnpk/must-gather-86wv8" event={"ID":"a24cd97c-0a48-402d-8d83-70e4d45ddc57","Type":"ContainerStarted","Data":"f3b94e7215a8529282f93220b362081865eb658e0d46063689b2384dd85ed0e0"} Sep 11 15:21:32.635229 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:32.635173 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-slnpk/must-gather-86wv8" podStartSLOduration=2.638402964 podStartE2EDuration="3.635157572s" totalImagesPullingTime="996.754608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:21:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:21:32.63288252 +0000 UTC m=+2483.876868180" watchObservedRunningTime="2026-09-11 15:21:32.635157572 +0000 UTC m=+2483.879143283" Sep 11 15:21:33.108633 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:33.108596 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mlrnk_a1b523f4-d191-49bb-8e0e-4999e6fa1a92/global-pull-secret-syncer/0.log" Sep 11 15:21:33.240150 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:33.240107 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vfmp4_ac306f32-44e2-4e36-8e09-2bb71e246eb8/konnectivity-agent/0.log" Sep 11 15:21:33.356770 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:33.356737 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-39.ec2.internal_34be09161f9ee3edbfcdbbe22b0b8213/haproxy/0.log" Sep 11 15:21:37.767730 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:37.767684 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-pxbzq_2264c0d5-92e6-432f-8e68-26de9c9c3b5a/manager/0.log" Sep 11 15:21:37.791516 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:37.791487 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-kl8dk_70c7f356-3f7a-4b55-b032-4165e1864109/kuadrant-console-plugin/0.log" Sep 11 15:21:37.932051 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:37.932020 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-gddvn_3d239b94-d5e2-47ee-9800-285917fcc7f4/manager/0.log" Sep 11 15:21:39.669094 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:39.669058 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6944cc9ff8-gkv7x_a8454816-3d65-4a4a-bb36-8b482cca02c3/metrics-server/0.log" Sep 11 15:21:39.861831 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:39.861804 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fp8dm_c8391c39-3072-4cba-9812-5dc45d82589d/node-exporter/0.log" Sep 11 15:21:39.882472 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:39.882435 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fp8dm_c8391c39-3072-4cba-9812-5dc45d82589d/kube-rbac-proxy/0.log" Sep 11 15:21:39.902315 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:39.902284 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fp8dm_c8391c39-3072-4cba-9812-5dc45d82589d/init-textfile/0.log" Sep 11 15:21:42.614757 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:42.614724 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lwrt2_57d4ba26-384d-42f8-86a1-335e27cdf762/console-operator/1.log" Sep 11 15:21:42.624129 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:42.624079 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-lwrt2_57d4ba26-384d-42f8-86a1-335e27cdf762/console-operator/2.log" Sep 11 15:21:43.274215 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:43.274179 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-slnpk/perf-node-gather-daemonset-jp2r8"] Sep 11 15:21:43.278078 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:43.278048 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-jp2r8" Sep 11 15:21:43.287961 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:43.287932 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-slnpk/perf-node-gather-daemonset-jp2r8"] Sep 11 15:21:43.361827 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:43.361789 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97358ff2-24d0-4120-9072-387f8f6570f1-sys\") pod \"perf-node-gather-daemonset-jp2r8\" (UID: \"97358ff2-24d0-4120-9072-387f8f6570f1\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-jp2r8" Sep 11 15:21:43.362107 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:43.362083 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/97358ff2-24d0-4120-9072-387f8f6570f1-podres\") pod \"perf-node-gather-daemonset-jp2r8\" (UID: \"97358ff2-24d0-4120-9072-387f8f6570f1\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-jp2r8" Sep 11 15:21:43.362260 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:43.362243 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/97358ff2-24d0-4120-9072-387f8f6570f1-proc\") pod \"perf-node-gather-daemonset-jp2r8\" (UID: \"97358ff2-24d0-4120-9072-387f8f6570f1\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-jp2r8" Sep 11 15:21:43.362380 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:43.362363 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97358ff2-24d0-4120-9072-387f8f6570f1-lib-modules\") pod \"perf-node-gather-daemonset-jp2r8\" (UID: \"97358ff2-24d0-4120-9072-387f8f6570f1\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-jp2r8" Sep 11 15:21:43.362522 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:43.362499 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmljd\" (UniqueName: \"kubernetes.io/projected/97358ff2-24d0-4120-9072-387f8f6570f1-kube-api-access-nmljd\") pod \"perf-node-gather-daemonset-jp2r8\" (UID: \"97358ff2-24d0-4120-9072-387f8f6570f1\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-jp2r8" Sep 11 15:21:43.463228 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:43.463190 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97358ff2-24d0-4120-9072-387f8f6570f1-sys\") pod \"perf-node-gather-daemonset-jp2r8\" (UID: \"97358ff2-24d0-4120-9072-387f8f6570f1\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-jp2r8" Sep 11 15:21:43.463400 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:43.463246 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/97358ff2-24d0-4120-9072-387f8f6570f1-podres\") pod \"perf-node-gather-daemonset-jp2r8\" (UID: \"97358ff2-24d0-4120-9072-387f8f6570f1\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-jp2r8" Sep 11 15:21:43.463400 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:43.463312 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/97358ff2-24d0-4120-9072-387f8f6570f1-proc\") pod \"perf-node-gather-daemonset-jp2r8\" (UID: \"97358ff2-24d0-4120-9072-387f8f6570f1\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-jp2r8" Sep 11 15:21:43.463400 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:43.463333 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97358ff2-24d0-4120-9072-387f8f6570f1-sys\") pod \"perf-node-gather-daemonset-jp2r8\" (UID: \"97358ff2-24d0-4120-9072-387f8f6570f1\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-jp2r8" Sep 11 15:21:43.463400 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:43.463339 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97358ff2-24d0-4120-9072-387f8f6570f1-lib-modules\") pod \"perf-node-gather-daemonset-jp2r8\" (UID: \"97358ff2-24d0-4120-9072-387f8f6570f1\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-jp2r8" Sep 11 15:21:43.463400 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:43.463387 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmljd\" (UniqueName: \"kubernetes.io/projected/97358ff2-24d0-4120-9072-387f8f6570f1-kube-api-access-nmljd\") pod \"perf-node-gather-daemonset-jp2r8\" (UID: \"97358ff2-24d0-4120-9072-387f8f6570f1\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-jp2r8" Sep 11 15:21:43.463602 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:43.463452 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97358ff2-24d0-4120-9072-387f8f6570f1-lib-modules\") pod \"perf-node-gather-daemonset-jp2r8\" (UID: \"97358ff2-24d0-4120-9072-387f8f6570f1\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-jp2r8" Sep 11 15:21:43.463602 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:43.463469 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/97358ff2-24d0-4120-9072-387f8f6570f1-podres\") pod \"perf-node-gather-daemonset-jp2r8\" (UID: \"97358ff2-24d0-4120-9072-387f8f6570f1\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-jp2r8" Sep 11 15:21:43.463602 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:43.463502 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/97358ff2-24d0-4120-9072-387f8f6570f1-proc\") pod \"perf-node-gather-daemonset-jp2r8\" (UID: \"97358ff2-24d0-4120-9072-387f8f6570f1\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-jp2r8" Sep 11 15:21:43.472147 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:43.472110 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmljd\" (UniqueName: \"kubernetes.io/projected/97358ff2-24d0-4120-9072-387f8f6570f1-kube-api-access-nmljd\") pod \"perf-node-gather-daemonset-jp2r8\" (UID: \"97358ff2-24d0-4120-9072-387f8f6570f1\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-jp2r8" Sep 11 15:21:43.596879 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:43.596807 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-jp2r8" Sep 11 15:21:43.882816 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:43.882783 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-slnpk/perf-node-gather-daemonset-jp2r8"] Sep 11 15:21:43.883838 ip-10-0-135-39 kubenswrapper[2583]: W0911 15:21:43.883803 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod97358ff2_24d0_4120_9072_387f8f6570f1.slice/crio-2e7f6e584d790322a019ac173c9b56ccc2c193cb99d01eed68b2d7d44b445243 WatchSource:0}: Error finding container 2e7f6e584d790322a019ac173c9b56ccc2c193cb99d01eed68b2d7d44b445243: Status 404 returned error can't find the container with id 2e7f6e584d790322a019ac173c9b56ccc2c193cb99d01eed68b2d7d44b445243 Sep 11 15:21:44.712774 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:44.712734 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-jp2r8" event={"ID":"97358ff2-24d0-4120-9072-387f8f6570f1","Type":"ContainerStarted","Data":"81374ac4efae26d8afa3796cec71186e8855becf743b7c5dbbe8a6dbeb6aa38e"} Sep 11 15:21:44.712978 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:44.712801 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-jp2r8" event={"ID":"97358ff2-24d0-4120-9072-387f8f6570f1","Type":"ContainerStarted","Data":"2e7f6e584d790322a019ac173c9b56ccc2c193cb99d01eed68b2d7d44b445243"} Sep 11 15:21:44.713289 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:44.713268 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-jp2r8" Sep 11 15:21:44.732158 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:44.732099 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-jp2r8" podStartSLOduration=1.732079363 podStartE2EDuration="1.732079363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:21:44.729010463 +0000 UTC m=+2495.972996120" watchObservedRunningTime="2026-09-11 15:21:44.732079363 +0000 UTC m=+2495.976065023" Sep 11 15:21:45.035940 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:45.035840 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7mrb4_70ec3434-0593-4acd-bd81-2580090a5020/dns/0.log" Sep 11 15:21:45.054430 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:45.054400 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7mrb4_70ec3434-0593-4acd-bd81-2580090a5020/kube-rbac-proxy/0.log" Sep 11 15:21:45.176635 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:45.176605 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ppnhx_fa59047c-1d81-44d0-b3a3-9e56bf53511c/dns-node-resolver/0.log" Sep 11 15:21:45.775872 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:45.775838 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4vlml_8dc4e60b-0baa-45d8-93b2-63436c43e3be/node-ca/0.log" Sep 11 15:21:46.692896 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:46.692859 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h_fe579da6-e7a4-4a75-aea1-c2957e757fa9/istio-proxy/0.log" Sep 11 15:21:47.587571 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:47.587515 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-rp6sn_b6775488-0433-4467-a80a-db83b0ca7de9/istio-proxy/0.log" Sep 11 15:21:47.593200 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:47.593169 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-rp6sn_b6775488-0433-4467-a80a-db83b0ca7de9/istio-proxy/1.log" Sep 11 15:21:47.636876 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:47.636844 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-xc4hx_b5e55d5d-7bdf-4a23-af47-fefd40e063d4/payload-pre-processing/0.log" Sep 11 15:21:48.598111 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:48.598062 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ht6zm_062d13b8-8971-4c91-b348-8b609ed42b56/serve-healthcheck-canary/0.log" Sep 11 15:21:49.205545 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:49.205517 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-th99j_0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b/kube-rbac-proxy/0.log" Sep 11 15:21:49.224766 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:49.224737 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-th99j_0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b/exporter/0.log" Sep 11 15:21:49.244942 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:49.244917 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-th99j_0fdaf3a0-5d6f-43e8-8746-6fd8698a7d9b/extractor/0.log" Sep 11 15:21:51.317038 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:51.317004 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-q9lp8_a9bb7c91-9123-4dc4-a469-7b7af81d9d4d/manager/0.log" Sep 11 15:21:51.735157 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:51.735127 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-jp2r8" Sep 11 15:21:51.906988 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:51.906962 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-s7722_ddc27eeb-1fbb-4110-a45d-c255ed11282c/server/0.log" Sep 11 15:21:52.054303 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:52.054190 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-r846f_ab41be90-a53b-4678-8c6b-9f3f8ea30dbf/manager/0.log" Sep 11 15:21:52.285125 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:52.285082 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67f987f856-vt2v2_63e3a376-f5d7-44f7-b725-e2f4ec532edc/manager/0.log" Sep 11 15:21:52.301079 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:52.301044 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67f987f856-vt2v2_63e3a376-f5d7-44f7-b725-e2f4ec532edc/copy-manifests/0.log" Sep 11 15:21:52.978931 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:52.978899 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h_fe579da6-e7a4-4a75-aea1-c2957e757fa9/istio-proxy/0.log" Sep 11 15:21:53.567121 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:53.567040 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6c79f8858d-vxvs5_10d337bb-1bc5-425f-8124-ab8afed6e1cc/manager/0.log" Sep 11 15:21:53.591531 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:53.591492 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-wcnb5_c3014ee9-77d1-4014-a146-e79bf0630cca/openshift-lws-operator/0.log" Sep 11 15:21:53.868233 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:53.868144 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-rp6sn_b6775488-0433-4467-a80a-db83b0ca7de9/istio-proxy/0.log" Sep 11 15:21:53.872982 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:53.872949 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-rp6sn_b6775488-0433-4467-a80a-db83b0ca7de9/istio-proxy/1.log" Sep 11 15:21:53.897508 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:53.897483 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-xc4hx_b5e55d5d-7bdf-4a23-af47-fefd40e063d4/payload-pre-processing/0.log" Sep 11 15:21:55.495837 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:55.495803 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799ctkm7h_fe579da6-e7a4-4a75-aea1-c2957e757fa9/istio-proxy/0.log" Sep 11 15:21:56.389564 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:56.389533 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-rp6sn_b6775488-0433-4467-a80a-db83b0ca7de9/istio-proxy/0.log" Sep 11 15:21:56.393313 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:56.393282 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-rp6sn_b6775488-0433-4467-a80a-db83b0ca7de9/istio-proxy/1.log" Sep 11 15:21:56.422018 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:56.421987 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-xc4hx_b5e55d5d-7bdf-4a23-af47-fefd40e063d4/payload-pre-processing/0.log" Sep 11 15:21:58.859675 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:58.859649 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-nzrr9_8da9b956-83ea-4049-80e3-e2682aec2402/migrator/0.log" Sep 11 15:21:58.877311 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:21:58.877280 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-nzrr9_8da9b956-83ea-4049-80e3-e2682aec2402/graceful-termination/0.log" Sep 11 15:22:00.282579 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:22:00.282545 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-97b8j_d39f63c1-32f1-4c2a-8585-a3defd284630/kube-multus/0.log" Sep 11 15:22:00.304961 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:22:00.304941 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-75zzz_c0b0394a-1326-4465-b674-23c0c6fbe017/kube-multus-additional-cni-plugins/0.log" Sep 11 15:22:00.323328 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:22:00.323303 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-75zzz_c0b0394a-1326-4465-b674-23c0c6fbe017/egress-router-binary-copy/0.log" Sep 11 15:22:00.341965 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:22:00.341934 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-75zzz_c0b0394a-1326-4465-b674-23c0c6fbe017/cni-plugins/0.log" Sep 11 15:22:00.359842 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:22:00.359820 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-75zzz_c0b0394a-1326-4465-b674-23c0c6fbe017/bond-cni-plugin/0.log" Sep 11 15:22:00.377680 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:22:00.377660 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-75zzz_c0b0394a-1326-4465-b674-23c0c6fbe017/routeoverride-cni/0.log" Sep 11 15:22:00.395460 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:22:00.395439 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-75zzz_c0b0394a-1326-4465-b674-23c0c6fbe017/whereabouts-cni-bincopy/0.log" Sep 11 15:22:00.412885 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:22:00.412863 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-75zzz_c0b0394a-1326-4465-b674-23c0c6fbe017/whereabouts-cni/0.log" Sep 11 15:22:00.801219 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:22:00.801191 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-brx7q_73182ce3-cb4b-4fcb-992e-85d116cf7e15/network-metrics-daemon/0.log" Sep 11 15:22:00.815078 ip-10-0-135-39 kubenswrapper[2583]: I0911 15:22:00.815053 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-brx7q_73182ce3-cb4b-4fcb-992e-85d116cf7e15/kube-rbac-proxy/0.log"